Writing a Screen Reader Smoke Test with Playwright

A screen reader smoke test written with Playwright drives a real or virtual screen reader through a scripted interaction and asserts the exact string it speaks. It prevents the single failure that automated DOM linters cannot see: an interaction that still produces valid markup but no longer announces anything to assistive technology. This guide shows the concrete @guidepup/playwright and virtual-screen-reader code to assert announcements for a sortable grid and a live region in continuous integration, and the headless caveats that decide which tool you can actually run where.

This is the implementation companion to screen reader smoke testing, which covers the surrounding loop — choosing journeys, baselining, and gating.

Spec reference

There is no W3C specification for “screen reader automation”; the contract you are testing against is the accessibility tree the browser computes from your ARIA, plus each screen reader’s rules for turning that tree into speech. Two open-source tools bridge Playwright to that speech:

  • @guidepup/playwright — a Playwright fixture that drives a real screen reader. On macOS it controls VoiceOver; on Windows it controls NVDA. It exposes the actual spoken log via lastSpokenPhrase() and spokenPhraseLog().
  • @guidepup/virtual-screen-reader — a screen reader simulator that walks the same accessibility tree the browser exposes and produces an equivalent spoken log, fully headless with no AT installed.

Default behaviour to know: a screen reader speaks in response to events — focus moves, activations, and live region mutations. If your interaction produces no accessibility-tree change, the spoken log is empty, and an empty log is exactly the regression a smoke test is designed to catch. The announcements you assert against are the ones you designed following screen reader announcement strategies.

The success criteria in scope are 4.1.3 Status Messages (Level AA) — the live region announcement — and 4.1.2 Name, Role, Value (Level A) — the control’s name and state in the sort announcement.

When to use real AT vs. the virtual screen reader

The choice is dictated by your CI environment and what fidelity you need, not preference.

Use @guidepup/virtual-screen-reader when you need a fast, always-on gate that runs on stock headless Linux CI. It has no OS or AT dependency, so it runs on every pull request in seconds. Its limitation is fidelity: it reflects the accessibility tree, not the idiosyncrasies of a specific screen reader version. It proves the tree is right, which catches the vast majority of “went silent” regressions.

Use @guidepup/playwright with real VoiceOver or NVDA when you need to confirm that an actual screen reader speaks what you expect, including vendor quirks the simulator cannot model. It requires a self-hosted macOS or Windows runner with the AT installed and automation permission granted, so it is slower and heavier — run it per release or nightly, not per pull request.

Do not try to run real AT on GitHub-hosted Linux runners (there is no screen reader to drive), and do not treat a green virtual-screen-reader run as proof of real-AT behaviour. They are layers, not substitutes.

Annotated code example

Real AT: assert a sort announcement under VoiceOver

// SC 4.1.2 name/role/value + SC 4.1.3 status message
// @guidepup/playwright provides a voiceOver fixture bound to real VoiceOver.
// Requires: macOS self-hosted runner + VoiceOver automation permission granted.
import { voTest as test, expect } from '@guidepup/playwright';

test('sorting the Amount column announces the new sort state', async ({ page, voiceOver }) => {
  await page.goto('/dashboard/transactions');

  // Move the VoiceOver cursor into the page's web content.
  await voiceOver.navigateToWebContent();

  // Jump to the sort button inside the Amount column header.
  // The button's accessible name comes from aria-label (WCAG 4.1.2).
  await voiceOver.perform(voiceOver.keyboardCommands.jumpToButton);

  // Activate it — equivalent to pressing Enter on the focused control.
  await voiceOver.act();

  // Assert the single announcement this interaction must produce.
  // This string is the committed baseline; any change fails the gate.
  expect(await voiceOver.lastSpokenPhrase()).toBe('Table sorted by Amount ascending');
});

Real AT: assert a live region announcement under NVDA

// SC 4.1.3 status message — the live region must speak the batched update once.
// On Windows the same test file swaps the voiceOver fixture for nvda.
import { nvdaTest as test, expect } from '@guidepup/playwright';

test('a streamed record announces politely, exactly once', async ({ page, nvda }) => {
  await page.goto('/dashboard/live');
  await nvda.navigateToWebContent();

  // Trigger one batched stream update from the test harness.
  await page.getByRole('button', { name: 'Emit one test record' }).click();

  // Wait on the CONDITION (region text present), not a fixed timeout,
  // so we do not race the polite announcement. See Gotcha 1.
  await page.locator('#stream-log').filter({ hasText: /new record/ }).waitFor();

  // The last spoken phrase is the polite region's announcement.
  expect(await nvda.lastSpokenPhrase()).toBe('1 new record — latest: AAPL 182.44');
});

Headless: the same journey under the virtual screen reader

// SC 4.1.3 status message — headless variant for per-pull-request gating.
// @guidepup/virtual-screen-reader needs no OS-level AT; runs on Linux CI.
import { test, expect } from '@playwright/test';
import { virtual } from '@guidepup/virtual-screen-reader';

test('grid sort announces (virtual SR, headless)', async ({ page }) => {
  await page.goto('/dashboard/transactions');

  // Point the virtual screen reader at the DOM container to observe.
  await virtual.start({ container: await page.locator('body').elementHandle() });

  await page.getByRole('button', { name: /amount/i }).focus();
  await virtual.press('Enter');                 // activate the sort
  await page.waitForFunction(() =>               // condition-based wait
    document.querySelector('#grid-announcements')?.textContent?.includes('sorted'));

  const log = await virtual.spokenPhraseLog();   // ordered spoken strings
  await virtual.stop();

  // The tree-derived phrasing differs slightly from real VoiceOver — keep a
  // per-adapter baseline rather than reusing the real-AT string verbatim.
  expect(log).toContain('Table sorted by Amount ascending');
});

Keyboard & AT behaviour

Interaction Real AT call (@guidepup/playwright) Virtual SR call Assertion target
Focus a control voiceOver.perform(cmd.jumpToButton) page.locator(...).focus() Name + role in phrase
Activate (Enter) voiceOver.act() virtual.press('Enter') Post-action announcement
Read last spoken string voiceOver.lastSpokenPhrase() last item of spokenPhraseLog() Single-announcement journeys
Read full ordered log voiceOver.spokenPhraseLog() virtual.spokenPhraseLog() Multi-step order and count
Wait for a polite region page.locator(id).waitFor() page.waitForFunction(...) Avoids racing the announcement

Integration context

This test is one journey inside the broader screen reader smoke testing loop: the strings you assert here are the baseline that loop defends release over release. The announcements themselves — their wording, politeness level, and timing — are a design decision made upstream in screen reader announcement strategies; a smoke test freezes that decision, it does not make it. When an assertion fails because the announcement genuinely changed for the better, update the baseline in the same reviewed pull request.

Gotchas

1. Racing a polite live region

A polite region speaks only after a pause, so lastSpokenPhrase() called immediately after the trigger reads whatever came before the update — often an empty string or a stale phrase. Always wait on a condition (the region’s textContent containing the expected substring) before reading the log, and allow the announcement batching interval to elapse. Fixed waitForTimeout values are the top cause of flaky screen reader tests.

2. Reusing one baseline across adapters

The virtual screen reader derives phrasing from the accessibility tree; real VoiceOver and NVDA apply their own verbosity rules. A string captured under one will not match the other exactly. Keep a separate baseline file per adapter and compare like with like, or your real-AT run will fail against a virtual-SR baseline for no real defect.

3. Automation permission not granted

@guidepup/playwright silently produces an empty spoken log if the OS has not granted the runner permission to control the screen reader. A whole suite of “expected ‘X’ got ‘’” failures usually means a permissions problem, not a product regression — verify the runner’s accessibility automation permission before debugging your ARIA.

FAQ

Can I run guidepup with a real screen reader on GitHub-hosted Linux runners?

No. Driving real VoiceOver or NVDA requires the operating system and screen reader to be installed and granted automation permission, so you need a self-hosted macOS or Windows runner. Stock Linux GitHub-hosted runners have no screen reader. For headless Linux CI, use @guidepup/virtual-screen-reader, which reads the browser’s accessibility tree without any AT installed. Run the real-AT suite on a nightly or per-release schedule against a self-hosted runner.

Should I assert the whole spoken log or just the last phrase?

Assert lastSpokenPhrase when a discrete interaction should produce exactly one announcement, such as activating a sort — it makes failures trivial to read. Assert the full spokenPhraseLog only when order and count across a multi-step journey matter, and normalise volatile substrings like numbers and timestamps before comparing so live data does not cause spurious diffs.

Why does my live region assertion pass locally but fail in CI?

A polite live region announces only after a speech pause, so a capture read too early records silence. Locally the machine is slower and the region has settled by the time you read it; in CI the read races the announcement. Wait on a condition — the region’s text content changing — rather than a fixed timeout, and allow the announcement batching interval to elapse before calling lastSpokenPhrase.

← Back to Screen Reader Smoke Testing