Building a Screen Reader Test Matrix

No team can test every screen reader with every browser on every release. A test matrix makes the trade-off explicit: which combinations are tested always, which periodically and which only when a feature needs it, and which features each combination must cover. For data UIs, the matrix also captures known differences — sort announcements, treegrid levels, live region timing — so testers know where to look and support teams can answer questions.

This guide belongs to screen reader smoke testing.

Spec reference

WCAG accessibility supported — techniques must work with the assistive technologies users have; the matrix documents what you verified.
WebAIM Screen Reader User Survey — public data on screen reader and browser combinations (NVDA and JAWS on Windows, VoiceOver on Apple devices, TalkBack on Android).
ACR / VPAT — conformance reports should state tested combinations.
Versions — screen readers and browsers update often; record versions.

Tier 1 covers most users every release.

When to use — and when not to

Build a matrix as soon as more than one person tests with screen readers or customers ask which assistive technologies you support.

Do not copy a matrix from another product without checking your own users; enterprise products may see far more JAWS, consumer products more VoiceOver iOS. The misapplication: a matrix listing ten combinations as “supported” when only one was ever tested.

Annotated code example

# a11y-matrix.yml — versioned with the product (versions below are examples)
tiers:
  1: { frequency: every-release }
  2: { frequency: monthly }
  3: { frequency: quarterly }
combinations:
  - id: nvda-chrome     ; tier: 1 ; at: NVDA 2026.1    ; browser: Chrome 140 ; os: Windows 11
  - id: jaws-chrome     ; tier: 1 ; at: JAWS 2026      ; browser: Chrome 140 ; os: Windows 11
  - id: vo-safari       ; tier: 1 ; at: VoiceOver      ; browser: Safari 19  ; os: macOS 16
  - id: vo-ios          ; tier: 2 ; at: VoiceOver      ; browser: Safari     ; os: iOS 19
  - id: talkback-chrome ; tier: 2 ; at: TalkBack 16    ; browser: Chrome     ; os: Android 16
features:
  - grid-navigation: [nvda-chrome, jaws-chrome, vo-safari]
  - sort-announcement: [nvda-chrome, jaws-chrome, vo-safari]
  - filter-count-announcement: [nvda-chrome, jaws-chrome, vo-safari, vo-ios, talkback-chrome]

Result record:
2026-09-28 | nvda-chrome | sort-announcement | PASS | "Amount, sorted ascending" | tester: QA-2
2026-09-28 | vo-safari   | sort-announcement | PASS with note | state read only on refocus; status message covers it

Keyboard & AT behaviour

Feature	Known differences to record
Grid navigation	Keys differ (NVDA Ctrl+Alt+Arrows, VoiceOver VO+Arrows)
Sort state	Some read aria-sort only on refocus
Treegrid levels	VoiceOver announces levels inconsistently
Live regions	Timing and dropping vary under load
Column headers	VoiceOver depends on table settings

The matrix is a living document.

Step-by-step implementation

Gather data. Use support tickets, customer surveys and public survey data; analytics cannot detect screen readers reliably and should not try.
Set tiers. Tier 1 for combinations most users have, tested every release; tier 2 monthly; tier 3 quarterly or on demand.
Map features. For each critical feature, list the combinations that must pass.
Record results. Date, versions, result, notes and tester.
Review quarterly. Update versions and tiers; retire combinations nobody uses.

Integration context

Automated coverage for tier 1 is described in NVDA smoke tests with Guidepup and VoiceOver smoke tests with Guidepup. Behaviour differences are collected in assistive technology behaviour differences, including Orca and Firefox on Linux. Conformance reporting is in VPAT and ACR for data-heavy products.

Gotchas

Detecting screen readers. Do not try to detect assistive technology in analytics; it is unreliable and raises privacy concerns.

Version drift. Browsers update monthly; record versions at test time rather than claiming a fixed version.

Licensing. JAWS requires a licence; budget for it if your users need it.

Edge cases

Enterprise customers. A large customer may standardise on one combination; add it to tier 1 for their release cycle.
Braille displays. Some users rely on braille output; include a braille check in tier 2 for critical paths.
Magnifier with speech. ZoomText Fusion combines both; consider it for workplace products.

Worked example

A B2B analytics vendor claimed “screen reader support” but tested only with VoiceOver on developers’ Macs. Customer complaints came from JAWS users in government agencies. The team surveyed customers, built a three-tier matrix with NVDA, JAWS and VoiceOver in tier 1, mapped grid and chart features to combinations, and recorded results per release. Their ACR now names tested combinations, and JAWS issues are caught before release.

Common mistakes

Untested “supported” claims.
Copying another product’s matrix.
No versions recorded.
Never revisiting tiers.

Testing checklist

Tiers are based on user data.
Critical features are mapped to combinations.
Results are recorded with versions and dates.
The matrix is reviewed quarterly.
The support statement and ACR reflect the matrix.

Use the matrix to plan a session: pick the tier 1 combinations, run each critical feature’s script and record results immediately in the matrix file. When a result differs between combinations, write the difference as a note, decide whether it needs a product fix (such as an extra status message) and link the issue. Over time, the notes become the team’s most valuable reference on screen reader behaviour.

Publish a short version of the matrix in your accessibility statement and give the full version to support and sales teams. Support can then answer “does it work with JAWS?” accurately, and sales can respond to procurement questionnaires without guessing. Keep the public version honest: list what you test and how often, and invite users of other combinations to report problems.

Budgeting time

A realistic tier 1 pass over five critical features in three combinations takes an experienced tester two to four hours. Plan that time into each release, automate the most stable paths with smoke tests, and spend the saved time on exploratory testing of new features rather than repeating scripts.

Key points

Tier combinations by real user data, not convenience.
Map critical features to the combinations that must pass.
Record versions, dates and notes with every result.
Review quarterly and publish an honest support statement.

Design system notes

Maintain a component-level matrix for the design system and let product matrices reference it, adding only product-specific features.

FAQ

How many combinations should tier 1 have?

Usually three: NVDA, JAWS and VoiceOver with their common browsers.

Where do I get user data?

Support tickets, customer surveys and public screen reader surveys.

Should mobile be tier 1?

If many users work on phones, yes.

How do I record results?

In a versioned file or test management tool with versions and dates.

How often should the matrix change?

Review quarterly; update versions each release.

Does the matrix go in the ACR?

The tested combinations should.

NVDA smoke tests with Guidepup — automated tier 1
Orca and Firefox on Linux with data grids — a tier 3 example
VPAT and ACR for data-heavy products — reporting

← Back to Screen Reader Smoke Testing