Screen Reader Smoke Testing

A screen reader smoke test is a small, fast, repeatable check that the handful of announcements your data UI absolutely depends on still fire, still say the right thing, and still fire in the right order. It is not a full audit. Its entire value is that it runs on every release without a human putting on headphones, so that the specific regressions automated accessibility linters cannot see — a sort that stops announcing, a filter count that goes silent, a live region that a framework re-render quietly detached — are caught before they reach production.

This page defines what a smoke test covers versus a full audit, shows how to build a repeatable per-release checklist for data grids, charts, and live regions, and walks through scripting assistive technology with Playwright and guidepup (or a virtual screen reader) so you can capture baseline announcement text and diff it release over release. It complements the linting layer described in automated accessibility testing pipelines: axe-core proves the DOM is well-formed; a smoke test proves the DOM is well-spoken.

WCAG Criteria in Scope

Criterion Level Relevance to smoke testing
1.3.1 Info and Relationships A Roles, headers, and positional metadata the smoke test asserts are announced
2.4.3 Focus Order A Focus landing points after sort, filter, and dialog close are part of each journey
4.1.2 Name, Role, Value A Control names and states (aria-sort, aria-pressed) must be present in captured speech
4.1.3 Status Messages AA Live region announcements — the most regression-prone strings — are the core assertion target

A smoke test does not replace conformance evaluation against these criteria; it guards the specific announcements that prove they are still met after a code change.

Smoke test vs. full audit

The distinction is one of scope, cadence, and cost, and confusing the two is the most common way teams waste effort.

Dimension Smoke test Full audit
Question answered “Did any critical announcement regress since last release?” “Does this product conform to WCAG 2.2 AA?”
Coverage 5–15 pre-chosen critical journeys Every page, state, and success criterion
Cadence Every pull request / every release Quarterly, or per major release
Executor Automated, in CI Human specialist with real AT
Output Pass/fail announcement diff Findings report with severity and remediation
Cost per run Seconds to minutes Days

A smoke test deliberately trades breadth for repeatability. It will never tell you your colour contrast is wrong or your form is confusing — that is audit territory. It tells you, cheaply and constantly, that the four announcements you already fixed have not silently broken again. Treat the two as layers: the audit establishes what “correct” sounds like; the smoke test freezes that as a baseline and defends it.

Prerequisites

This page assumes you can:

The smoke-test loop

Every durable smoke suite is the same five-stage loop. You define a small set of critical journeys, capture what each one should announce, replay them on each release, diff the fresh capture against the committed baseline, and gate the merge on the result. A failing diff either reveals a real regression or a legitimate copy change — in which case you review it and re-baseline.

Screen reader smoke-test loop A left-to-right pipeline of five stages: define critical journeys, capture expected announcements, run per release, diff against baseline, and gate the merge. A pass path exits to ship. A fail path loops back from the gate to re-review and re-baseline the expected announcements. Define critical journeys Capture expected announcements Run per release (Playwright + AT) Diff vs baseline string compare Gate merge pass / fail PASS → ship FAIL → review the diff: real regression, or intended copy change → re-baseline Per-release announcement guard (WCAG 4.1.3) Small journey set · deterministic replay · string-level diff

The automation stack, layer by layer

A smoke test needs three cooperating layers. Understanding what each is responsible for prevents the common mistake of asserting the DOM when you mean to assert the speech.

The driver — Playwright

Playwright launches the browser, navigates to the route, and performs the keyboard interactions that constitute a journey (Tab, Enter, ArrowDown). It is the same tool many teams already use for end-to-end tests, so the smoke suite reuses existing fixtures and CI wiring.

The screen reader adapter — guidepup or a virtual screen reader

Playwright cannot hear anything. To capture what is spoken you need one of two adapters:

  • guidepup / @guidepup/playwright drives a real screen reader — VoiceOver on macOS or NVDA on Windows — and returns the actual spoken log. Highest fidelity, but requires a self-hosted runner with the OS and AT installed, and the AT’s automation permissions granted.
  • @guidepup/virtual-screen-reader is a screen reader simulator that reads the same accessibility tree the browser exposes, entirely headless. Lower fidelity (it does not reproduce every NVDA or JAWS quirk), but it runs on any Linux CI runner with no AT installed and is fast enough for per-PR gating.

The pragmatic split: run the virtual screen reader on every pull request as the always-on gate, and run real AT via guidepup on a nightly or per-release schedule to catch the vendor-specific deviations the simulator cannot model.

The assertion — announcement string comparison

Whichever adapter you use, the output is an ordered list of spoken strings. The smoke test asserts that list against a committed baseline. Because the assertion is on text, not pixels, it is stable, reviewable in a pull request diff, and immune to visual-only churn.

Step-by-Step Implementation

Step 1 — Enumerate the critical journeys (WCAG 4.1.3, 2.4.3)

A journey is the smallest sequence of interactions that produces an announcement your users depend on. Keep the set small — five to fifteen — and choose the ones whose silent failure would be worst. For a data grid the canonical three are: sort a column, filter to a subset, and receive a streamed update.

// A journey is data, not code: name it, describe the steps, and
// state the announcement contract it must satisfy.
// SC 4.1.3 status message + SC 2.4.3 focus order are the guarded behaviours.
export const criticalJourneys = [
  {
    id: 'grid-sort',
    name: 'Sorting a column announces the new sort state',
    route: '/dashboard/transactions',
    // The exact spoken string(s) this journey must still produce.
    expects: ['Table sorted by Amount ascending'],
  },
  {
    id: 'grid-filter',
    name: 'Filtering announces the visible row count',
    route: '/dashboard/transactions',
    expects: ['12 rows shown'],
  },
  {
    id: 'live-stream',
    name: 'A streamed record announces once, politely',
    route: '/dashboard/live',
    expects: ['1 new record — latest: AAPL 182.44'],
  },
];

Step 2 — Capture the baseline announcement log (WCAG 4.1.3)

Run each journey once against a known-good build and record the spoken log verbatim. Commit it to the repository as the source of truth. This is a manual, reviewed step — a human confirms the strings are actually correct before they become the baseline everything else is measured against.

// SC 4.1.3: the spoken log is the artifact we defend release over release.
import { voStore } from '@guidepup/virtual-screen-reader';

async function captureBaseline(page, journey) {
  await page.goto(journey.route);
  const vsr = await voStore.start({ container: await page.$('body') });

  await page.getByRole('button', { name: /amount/i }).press('Enter');
  await page.waitForTimeout(400);           // let the polite region settle

  const spokenLog = await vsr.spokenPhraseLog();  // ordered array of strings
  await vsr.stop();
  return spokenLog;                          // write to baselines/grid-sort.json
}

Step 3 — Replay the journey on each release (WCAG 4.1.2, 4.1.3)

The per-release run is the baseline capture with the recording swapped for an assertion source. It performs the identical keystrokes and collects the identical log shape, so the only variable is the code under test.

// SC 4.1.2 name/role/value + SC 4.1.3 status message: replay the journey
// and collect the fresh spoken log for diffing.
import { test } from '@playwright/test';
import { virtual } from '@guidepup/virtual-screen-reader';

test('grid-sort announces sort state', async ({ page }) => {
  await page.goto('/dashboard/transactions');
  await virtual.start({ container: await page.locator('body').elementHandle() });

  // Tab to the Amount header's sort button, then activate it.
  await page.getByRole('button', { name: /amount/i }).focus();
  await virtual.press('Enter');
  await page.waitForTimeout(400);

  const fresh = await virtual.spokenPhraseLog();
  await virtual.stop();
  // fresh is compared against the committed baseline in Step 4.
  return fresh;
});

Step 4 — Diff fresh capture against the baseline (WCAG 4.1.3)

The diff is a plain ordered-string comparison. Report added, removed, and changed lines. A non-empty diff fails the test; a reviewer then decides whether it is a regression to fix or an intended copy change to re-baseline.

// SC 4.1.3: any change to the announcement log must be reviewed, never silent.
import { readFileSync } from 'node:fs';

function diffAnnouncements(baselinePath, fresh) {
  const baseline = JSON.parse(readFileSync(baselinePath, 'utf8'));
  const changes = [];
  const max = Math.max(baseline.length, fresh.length);

  for (let i = 0; i < max; i++) {
    if (baseline[i] !== fresh[i]) {
      changes.push({ index: i, expected: baseline[i] ?? null, actual: fresh[i] ?? null });
    }
  }
  return changes;   // empty array === pass
}

Step 5 — Gate the merge on an unreviewed diff (WCAG 4.1.3)

Wire the diff into the test assertion so a changed announcement blocks the pipeline. The failure message must print both strings — a bare “assertion failed” wastes the reviewer’s time.

// SC 4.1.3: fail the build on any un-baselined announcement change.
import { expect, test } from '@playwright/test';

test('grid-sort matches announcement baseline', async ({ page }) => {
  const fresh = await runJourney(page, criticalJourneys[0]);   // Steps 1–3
  const changes = diffAnnouncements('baselines/grid-sort.json', fresh);   // Step 4

  expect(
    changes,
    changes.map(c => `#${c.index}: expected "${c.expected}" got "${c.actual}"`).join('\n')
  ).toEqual([]);
});

Keyboard Interaction Contract

Each journey is expressed as keystrokes. The smoke test replays these exactly; the contract below is what the assertions encode.

Key Context Action Expected AT announcement Failure indicator
Tab Grid header Focus the sort button “Amount, not sorted, button” Focus skips the button; log is empty
Enter Sort button Toggle sort ascending “Amount, sorted ascending” + “Table sorted by Amount ascending” Icon changes but log is unchanged
Typed query Filter input Filter rows, debounced “12 rows shown” Count missing or announced per keystroke
(automatic) Live feed One batched update fires “1 new record — latest: AAPL 182.44” Per-record flooding or silence
Escape Open dialog Close and restore focus Trigger control name re-announced Focus lost to body; no announcement

Screen Reader Compatibility Matrix

Announcement text is not portable across AT. The baseline you capture with a virtual screen reader will differ from real NVDA, which differs from VoiceOver. Maintain a baseline per adapter rather than assuming one string set fits all.

AT + Browser Automatable via Fidelity Notes for smoke testing
Virtual screen reader (headless) @guidepup/virtual-screen-reader Accessibility-tree only Runs on any Linux CI runner; ideal per-PR gate; will not reproduce vendor quirks
NVDA 2024 + Chrome/Firefox @guidepup/playwright Real speech log Windows self-hosted runner; announces aria-sort and polite regions reliably
VoiceOver + Safari (macOS) @guidepup/playwright Real speech log macOS runner; requires automation permission; suppresses some polite updates mid-speech
JAWS 2024 + Chrome No first-class Playwright driver Manual / vendor tooling Keep in the periodic full audit, not the automated smoke gate
TalkBack + Chrome (Android) No headless driver Manual Touch-exploration journeys are audit-only; do not attempt to gate on them

Edge Cases & Failure Modes

1. Flaky timing on polite regions

A polite live region announces only after a speech pause, so a capture taken too early records silence and fails spuriously. Fix: wait on a condition (the region’s text content changing) rather than a fixed millisecond timeout, and give the batching interval described in real-time data stream announcements time to settle before reading the log.

2. Headless environments have no real AT

The virtual screen reader reads the accessibility tree the browser computes, which is close to but not identical to what NVDA or VoiceOver speak. Never treat a passing virtual-SR run as proof of real-AT behaviour — it proves the tree is right, not the speech. Keep at least one real-AT run in the release pipeline.

3. Non-deterministic data breaks string matching

If the grid renders live values (“AAPL 182.44”), the announcement string changes every run and every diff fails. Fix: seed the route with fixture data for smoke runs, or normalise volatile substrings (numbers, timestamps) to placeholders before diffing.

4. Framework re-render detaches the live region

The failure a smoke test exists to catch: a refactor moves the live region inside a component that unmounts on route change, so the AT stops observing it and every announcement goes silent. Automated linters see a valid role="status" element and pass; only the announcement diff catches it. This is why the live-region journey is non-negotiable in the suite.

5. Baseline drift from unreviewed re-baselining

If re-baselining is a one-command shortcut with no review, developers will silence real regressions by regenerating the baseline. Fix: require the baseline file change to appear in the pull request diff and be approved like any other code change.

Deep dive: writing the Playwright test

The end-to-end mechanics — installing @guidepup/playwright, granting AT automation permissions, driving VoiceOver and NVDA, and asserting announcement strings for a sortable grid and a live region, including the headless caveats — are covered in the dedicated guide: writing a screen reader smoke test with Playwright.

Key facts from that page:

  • @guidepup/playwright extends Playwright’s fixture with a voiceOver / nvda handle whose .lastSpokenPhrase() and .spokenPhraseLog() return the real speech log.
  • Real AT drivers require a self-hosted runner with the OS and screen reader installed; they will not run on stock Linux GitHub-hosted runners.
  • Assert on the last spoken phrase after a discrete interaction, not the whole log, when a journey should produce exactly one announcement — it makes failures far easier to read.
// Minimal shape — full annotated version in the deep-dive guide.
// SC 4.1.3: assert the single announcement a sort must produce.
import { voTest as test, expect } from '@guidepup/playwright';

test('sort announces once under real VoiceOver', async ({ page, voiceOver }) => {
  await page.goto('/dashboard/transactions');
  await voiceOver.navigateToWebContent();
  await voiceOver.perform(voiceOver.keyboardCommands.jumpToButton); // Amount sort
  await voiceOver.act(); // Enter
  expect(await voiceOver.lastSpokenPhrase()).toBe('Table sorted by Amount ascending');
});

Testing Checklist

Suite setup

Per-run behaviour

Coverage

← Back to Testing & Auditing Accessible Data Interfaces