Usability Testing with Screen Reader Users

Conformance testing tells you whether a grid meets WCAG. Usability testing with screen reader users tells you whether people can actually get their work done with it — and the two often disagree. A grid can pass every check yet be exhausting to use: every cell announced with five properties, sort feedback that arrives after the user has moved on, a filter flow with twelve steps. The only way to find that out is to watch experienced screen reader users try real tasks.

This guide belongs to accessibility governance for data products.

Spec reference

WCAG does not require usability testing, but its understanding documents repeatedly note that conformance does not guarantee usability for every user. Relevant practice standards:

ISO 9241-11 — usability as effectiveness, efficiency and satisfaction in a context of use.
WAI “Involving Users in Evaluating Web Accessibility” — guidance on including people with disabilities in evaluation.
Ethical research practice: informed consent, fair compensation, accessible consent forms and data protection.

Small studies of five to eight participants find most major problems.

When to use — and when not to

Run sessions before major releases of data-heavy features, when redesigning core workflows, and periodically (twice a year) on the most-used grids and dashboards. Use them to evaluate alternatives — for example, two ways of announcing filter results.

Do not use usability sessions as your first accessibility check; fix conformance issues found by audits and automated testing first, or sessions will be spent on known basics. Do not test with only one participant and generalise; screen reader users differ widely in tools, settings and expertise.

The misapplication: asking a single screen reader user, often a colleague, to “check whether it’s accessible” in ten minutes, without tasks or compensation, and treating their opinion as a sign-off.

Annotated code example

A session plan for a data grid, stored with the study so it can be repeated:

study: invoices-grid-q4
participants: 6
screen_readers: [NVDA, JAWS, VoiceOver]       # mix of tools and expertise
environment: https://usability.example.com     # stable build, test data, no real customer data
tasks:
  - id: T1
    goal: "Find invoices from Lund & Co that are overdue and note the largest amount."
    success: "States 1,240.00 within 5 minutes"
    observe: [filter usage, sort usage, table navigation commands, announcements heard]
  - id: T2
    goal: "Mark the three oldest overdue invoices as disputed."
    success: "Three correct rows updated"
    observe: [selection model, bulk action discovery, confirmation messages]
  - id: T3
    goal: "Export the overdue invoices and open them in your usual spreadsheet."
    success: "File opens with headers announced"
consent: accessible-consent-v3.html             # screen reader tested
compensation: "Paid at expert consultant rate, in advance of the session"

// Lightweight instrumentation for the test build: log key interactions with timestamps (no personal data)
const log = (event, detail) => navigator.sendBeacon('/usability-log', JSON.stringify({ t: Date.now(), event, detail }));
grid.addEventListener('keydown', (e) => log('key', { key: e.key, role: e.target.getAttribute('role') }));

Keyboard & AT behaviour

What to observe and record during sessions:

Observation	Why it matters
Navigation commands used (table commands, arrows, headings)	Shows whether your structure supports users’ strategies
Announcements users act on — and ones they interrupt	Reveals verbosity and timing problems
Points where users switch strategy or give up	Pinpoints usability failures
Time per task compared with sighted baseline	Quantifies efficiency gaps
Users’ own settings (verbosity, punctuation)	Explains differences between participants

Most of what sessions find, automated tests cannot.

Integration context

Before running sessions, make sure the basics are right with manual accessibility audits and screen reader smoke testing. Findings about verbosity often lead to verbosity settings for data announcements. Turn findings into criteria using writing accessibility acceptance criteria for data grids.

Gotchas

Remote audio. Participants’ screen reader speech must reach you. Use a conferencing tool with computer audio sharing, and test it in a pre-session call.

Speech rate. Experienced users listen at high speeds; ask whether they are comfortable slowing down for the recording, but never insist.

Unfamiliar test environments. Avoid environments that require new software installs or unusual logins; access problems eat session time.

Common mistakes

Testing before fixing known conformance issues.
One participant treated as representative.
Unpaid or underpaid participants.
Leading questions (“Did you hear the sort announcement?”).

Framework notes

Usability studies are framework-agnostic. Practically, give participants a stable build deployed with feature flags fixed, test data that resembles real data in size and shape (a 12-row grid will not reveal problems a 2,000-row grid does), and no analytics or chat widgets that interrupt screen readers.

Worked example

A finance product ran six remote sessions with NVDA, JAWS and VoiceOver users on its invoice grid. All tasks were completed, but the median time for “find overdue invoices from Lund & Co” was four times the sighted baseline. Recordings showed why: each cell announced status, owner and due-date descriptions, and filter results were announced after users had already started navigating. The team cut per-cell descriptions to a verbosity setting, moved the filter announcement earlier, and re-tested with three participants: task time halved.

Testing checklist

Participants are daily screen reader users with a mix of tools and expertise.
Compensation is fair and paid promptly.
Tasks are realistic with clear success criteria.
Sessions capture both screen and screen reader audio.
Findings are prioritised and tracked to fixes and re-tests.

Design system notes

Feed findings about component behaviour (verbosity, timing, keyboard models) back into the design system, so improvements reach every product. Keep a summary of past studies in the design system documentation.

FAQ

How many participants do we need?

Five to eight per round usually reveals most major problems; include a mix of screen readers and expertise levels.

Where do we recruit?

Disability organisations, specialised recruiting agencies, accessibility communities and your own customers who use assistive technology.

Should sessions be remote or in person?

Remote sessions let participants use their own configured setups, which usually gives better results.

How much should participants be paid?

At least the rate you pay other expert participants; many teams pay a premium for specialised recruitment.

Can we test with sighted developers using screen readers?

That is useful for smoke testing, but it is not usability testing; developers’ strategies differ fundamentally from daily users’.

How do we measure success?

Task completion, time on task, errors and participants’ own ratings, compared across rounds.

Acceptance criteria for data grids — turning findings into requirements
Screen reader smoke testing — the developer-level check
Verbosity settings for data announcements — a common outcome

← Back to Accessibility Governance for Data Products