echokit

Chrome Extension Automated Testing for CI/CD - Research & Recommendations

Executive Summary

Goal: Implement automated regression testing for EchoKit Chrome extension in CI/CD pipeline that blocks PR merges to develop unless all tests pass.

Current State:

Recommended Solution: Multi-tiered approach combining headless-compatible tests with strategic use of headed mode in CI.

🔍 Research Findings (2026)

1. Testing Frameworks for Chrome Extensions

Playwright (Recommended ⭐)

Puppeteer

Selenium

Verdict: Stick with Playwright - EchoKit already uses it, and it has the best modern extension support.

2. Headless Mode Challenges & Solutions

The Problem

Chrome extensions with service workers have historically struggled in headless mode:

The Solutions (2026)

Option A: New Chromium Channel (Preferred)

context = playwright.chromium.launch_persistent_context(
    user_data_dir,
    channel='chromium',  # Key: Uses newer headless implementation
    args=[
        '--disable-extensions-except=/path/to/extension',
        '--load-extension=/path/to/extension'
    ]
)

✅ Works in CI
✅ No display server needed
✅ Official Playwright recommendation

Option B: Xvfb (Virtual Display Server)

- name: Run tests
  run: xvfb-run -a python3 tests/smoke_echokit.py

✅ Works reliably
⚠️ Requires Linux (Ubuntu/Debian)
⚠️ Slightly slower than pure headless

Option C: GitHub Actions with Headed Mode

- uses: browser-actions/setup-chrome@latest
- run: python3 tests/smoke_echokit.py
  env:
    DISPLAY: ':99'

Requires additional setup but most reliable.

3. Testing Strategy: Test Pyramid

Following industry best practices (Chrome Extension Testing Guide 2026):

        /\
       /10\    E2E Tests (Playwright - Full browser)
      /____\   
     /      \
    /   20%  \  Integration Tests (API + State)
   /__________\
  /            \
 /     70%      \ Unit Tests (Jest/Vitest - Fast)
/________________\

Tier 1: Unit Tests (70%) - NEW

Tier 2: Integration Tests (20%) - NEW

Tier 3: E2E Tests (10%) - EXISTING (Enhanced)

4. Visual Regression Testing

Prevent “blank screen” bugs that slip through functional tests.

Option A: Percy.io (Cloud SaaS)

from percy import percy_snapshot
percy_snapshot(browser, "EchoKit Popup - Default State")

✅ Easy setup, hosted comparison
✅ Works in CI out of box
✅ PR comments with visual diffs
❌ Costs $249+/month for private repos

Option B: Chromatic (Storybook-focused)

Option C: Playwright Built-in

await page.screenshot(path='baseline.png')
assert compare_images('baseline.png', 'current.png', threshold=0.01)

✅ Free and self-hosted
✅ Full control over baselines
❌ Manual baseline management
❌ No automatic PR comments

Recommendation: Start with Playwright built-in visual comparison (free), evaluate Percy after 3 months if needed.

📋 Recommended Implementation Plan

Phase 1: Fix CI/CD Pipeline (Week 1) ⭐ PRIORITY

Objective: Get existing smoke tests running in CI

Tasks:

  1. Update tests/smoke_echokit.py to use channel='chromium'
  2. Add fallback to xvfb if chromium channel fails
  3. Enable smoke test job in .github/workflows/test.yml
  4. Add branch protection rule: require smoke job to pass
  5. Test on a feature branch → PR → verify tests block merge

Acceptance Criteria:

Files to Modify:

Phase 2: Add Unit Tests (Week 2-3)

Objective: 70% test coverage with fast unit tests

Tasks:

  1. Set up Vitest (modern, fast, ESM-native)
  2. Write tests for core modules:
    • shared/matcher.js - 20 test cases
    • shared/app.js - state management tests
    • shared/json-highlight.js - formatting tests
  3. Add to CI pipeline (runs before smoke tests)
  4. Aim for 70%+ coverage

New Files:

Phase 3: Visual Regression (Week 4)

Objective: Catch UI regressions automatically

Tasks:

  1. Add screenshot capture to smoke tests
  2. Store baseline screenshots in tests/baselines/
  3. Compare screenshots on each test run
  4. Fail test if diff > 2% pixels
  5. Add baseline update command for intentional UI changes

Implementation:

# In smoke_echokit.py
def test_popup_renders():
    popup_page.goto(popup_url)
    popup_page.wait_for_selector('[data-testid="echokit-app"]')
    
    # Visual regression check
    actual = popup_page.screenshot()
    baseline = Path('tests/baselines/popup-default.png')
    diff = compare_images(baseline, actual)
    assert diff < 0.02, f"Visual diff {diff:.2%} exceeds threshold"

Phase 4: Integration Tests (Future)

Objective: Test message passing and Chrome API interactions

Tasks:

🔧 Immediate Action Items

1. Update tests/smoke_echokit.py

Change Required:

# Current (line ~100-120):
context = p.chromium.launch_persistent_context(
    "",
    headless=False,  # ❌ Doesn't work in CI
    args=[...]
)

# Updated:
context = p.chromium.launch_persistent_context(
    "",
    channel='chromium',  # ✅ Enables headless extension support
    headless=True,       # ✅ Works in CI
    args=[...]
)

2. Update .github/workflows/test.yml

Remove the if: false gate:

smoke:
  runs-on: ubuntu-latest
  if: false  # ❌ Remove this line

Update to:

smoke:
  runs-on: ubuntu-latest
  steps:
    - uses: actions/checkout@v4
    - uses: actions/setup-python@v5
      with: { python-version: '3.11' }

    - name: Install Playwright
      run: |
        pip install playwright
        python -m playwright install chromium

    - name: Run smoke tests
      run: python3 tests/smoke_echokit.py

3. Add Branch Protection Rule

GitHub Settings → Branches → Branch protection rules:

Rule for develop:

🎯 Success Metrics

Track these to measure testing effectiveness:

Metric Current Target (1 month)
Tests passing in CI 0% (disabled) 100%
Test runtime in CI N/A < 2 minutes
PRs blocked by tests 0 All failing PRs
Blank screen bugs in prod 1+ 0
Test coverage ~5% (E2E only) 40%+ (unit + E2E)
Time to debug failures Hours < 10 minutes

📚 Additional Resources

Official Documentation

Community Resources

Testing Tools

🚨 Known Gotchas & Troubleshooting

Issue 1: “Failed to get extension ID”

Cause: Extension doesn’t have a service worker Fix: Ensure manifest.json includes:

"background": {
  "service_worker": "background.js"
}

Issue 2: Service worker timeout in CI

Cause: Headless mode doesn’t support extensions (old headless) Fix: Use channel: 'chromium' in launch config

Issue 3: Extension loads but popup is blank

Cause: CSP violations or missing resources Fix:

  1. Check browser console for errors
  2. Verify all resources in web_accessible_resources
  3. Use headless: False locally to debug

Issue 4: Tests flaky in CI but work locally

Cause: Race conditions, timing issues Fix:

  1. Use explicit waits: page.wait_for_selector() instead of time.sleep()
  2. Increase timeout: timeout=10000 (10 seconds)
  3. Wait for service worker: context.wait_for_event('serviceworker')

Issue 5: Tests pass but PR still merges

Cause: Branch protection not configured Fix: Enable required status checks in GitHub settings

🏁 Quick Start Guide

To run tests locally:

# Install dependencies
pip install playwright
python -m playwright install chromium

# Run all tests
python3 tests/smoke_echokit.py

# Run with headed browser (for debugging)
HEADLESS=false python3 tests/smoke_echokit.py

To add a new test case:

# In tests/smoke_echokit.py

def step(name, ok, detail=''):
    # ... existing code ...

# Add your test
resp = popup_page.evaluate("window.doFetch('/api/new-endpoint')")
step("NEW: New endpoint captured",
     resp['status'] == 200 and 'expected_field' in resp['body'])

To update visual baselines (after intentional UI changes):

# Take new baseline screenshots
python3 tests/smoke_echokit.py --update-baselines

# Review changes
git diff tests/baselines/

# Commit new baselines
git add tests/baselines/
git commit -m "chore: update visual test baselines for new UI"

🎬 Next Steps

  1. Immediate (Today):
    • Review this document with team
    • Decide on Phase 1 timeline
    • Assign owner for CI/CD fixes
  2. This Week:
    • Implement Phase 1 (fix CI pipeline)
    • Test on feature branch
    • Enable branch protection
  3. This Month:
    • Implement Phase 2 (unit tests)
    • Reach 40%+ test coverage
    • Zero blank screen bugs
  4. This Quarter:
    • Implement Phase 3 (visual regression)
    • Implement Phase 4 (integration tests)
    • Achieve 70%+ test coverage

📞 Questions?

Q: Will this slow down our development velocity? A: Initially ~10 minutes to set up, then saves hours debugging production issues. Net positive after first week.

Q: What if tests are flaky? A: Start with deterministic tests (syntax, structure). Add E2E tests carefully with proper waits. Aim for <1% flake rate.

Q: Can we skip tests for hotfixes? A: No. Tests are especially important for hotfixes. Use feature flags to disable risky features instead.

Q: What about manual testing? A: Automated tests complement (not replace) manual testing. Still manually test before releases.

Q: How do we handle breaking changes to the test suite? A: Update tests in same PR as code changes. Never merge broken tests to develop.

Last Updated: 2026-05-12 Document Owner: Development Team Related Docs: AUTOMATED_TESTING.md, CONTRIBUTING.md, .github/workflows/test.yml