Tracking Violation Trends Over Time

A single axe scan tells you what is wrong today. Tracking scans over time tells you whether things are getting better, which teams introduce regressions and which rules keep recurring. Done poorly, trends become vanity metrics: a falling number that reflects pages removed from the scan rather than issues fixed. Done well, they drive prioritisation and catch regressions the day they ship.

This guide belongs to automated accessibility testing pipelines.

Spec reference

  • axe-core results — violations with rule ID, impact, nodes and help URL.
  • WCAG mapping — axe tags (wcag2a, wcag2aa, wcag22aa) map rules to criteria.
  • Coverage — automated tools find roughly a third to a half of WCAG issues; trends cover only those.
  • Governance — trend data feeds accessibility reviews and ACRs.
A quarter of violation tracking Timeline: in week 1 the baseline is recorded; week 3 a regression spike from a new grid release; week 4 the regression is fixed; weeks 6 to 12 backlog falls steadily as teams fix rules by priority. A quarter of violation trackingWeek 1baseline: 412 violations across 60 statesWeek 3spike: +38 from new grid releaseWeek 4regression fixed within a weekWeek 8backlog down to 290Week 12backlog down to 160; colour contrast rule cleared
Spikes are regressions; slopes are backlog work.

When to use — and when not to

Track trends once automated scans run regularly across a stable set of pages and states. Review them in team rituals, not just in a dashboard nobody opens.

Do not use raw totals as the headline metric; they change when you add or remove pages. The misapplication: celebrating a drop from 400 to 200 violations that happened because half the pages were removed from the scan list.

Vanity metrics versus useful metrics Comparison: raw totals are distorted by coverage changes; normalised counts, regressions per release and time to fix reflect real progress. Vanity metrics versus useful metrics✗ Raw violation total"Down 50%!"No distinction of new issuesCounts only✓ Violations per scanned stateSame coverage, real changeNew regressions per releaseTime to fix regressions

Annotated code example

// After each Playwright + axe run, store a compact record
import { AxeBuilder } from '@axe-core/playwright';
const results = await new AxeBuilder({ page }).withTags(['wcag2a', 'wcag2aa', 'wcag22aa']).analyze();
const rows = results.violations.flatMap(v => v.nodes.map(n => ({
  run: process.env.GITHUB_RUN_ID, sha: process.env.GITHUB_SHA, date: new Date().toISOString(),
  page: pageId, state: stateId, rule: v.id, impact: v.impact,
  target: n.target.join(' '), fingerprint: hash(v.id + pageId + stateId + n.target.join(' ')),   // stable identity
})));
await appendToStore(rows);   // e.g. a table in a warehouse, or JSON artifacts aggregated nightly
-- New regressions in this run: fingerprints not seen in the previous main-branch run
SELECT rule, page, state, target FROM violations v
WHERE run = :current AND fingerprint NOT IN (SELECT fingerprint FROM violations WHERE run = :previous_main);

Keyboard & AT behaviour

Metric What it tells you
Violations per scanned state Density, comparable across coverage changes
New fingerprints per release Regressions introduced
Median days to fix regressions Team responsiveness
Top rules by count Where systemic fixes pay off
Manual findings per audit What automation misses
Violations by rule at quarter end Bar chart of remaining violations by rule: colour contrast 0, button name 22, ARIA required children 41, label 18, focus order semantics 9. Violations by rule at quarter endcolor-contrastviolations (cleared)button-nameviolationsaria-required-childrenviolationslabelviolationsscrollable-region-focusableviolations
Rule-level views guide systemic fixes.

Step-by-step implementation

  1. Fix coverage. Define the list of pages and states scanned (including grid states: filtered, editing, expanded) and version it.
  2. Fingerprint violations. Combine rule, page, state and target selector into a stable identity, so you can tell new from existing.
  3. Store per run. Append results to a store keyed by run and commit.
  4. Compute metrics. Density per state, new regressions per release, days to fix, top rules.
  5. Review regularly. A short weekly look by each team; a monthly review with product leadership.

Integration context

Scans are set up as in setting up axe-core in a GitHub Actions pipeline and pa11y-ci for multi-state data pages. Grid states are covered in axe scans of grid states with Playwright. Governance use of the data is in accessibility regression triage for data UIs and tracking accessibility debt in component libraries.

Gotchas

Selector instability. Generated class names change between builds and break fingerprints; prefer stable attributes or roles in targets, or fingerprint on rule plus page plus state plus element role and name.

Rule changes in axe updates. Upgrading axe can add or change rules; annotate the dashboard when you upgrade.

One component, many violations. A broken shared component appears on every page; group by component to avoid inflated counts.

Edge cases

  • Flaky states. States that sometimes fail to render produce noisy trends; stabilise them before tracking.
  • Feature flags. Record flag states with runs, or trends jump when flags flip.
  • Multiple products. Normalise per product; do not compare raw totals across products.

Worked example

A data platform team reported a falling violation count each month, but users kept reporting the same problems. Investigation showed pages were being dropped from the scan list as they failed. The team fixed the coverage list, added fingerprints, tracked new regressions per release and time to fix, and grouped violations by component. The real picture — a few shared components causing most violations — led to targeted fixes that cut density by 60% in a quarter.

Common mistakes

  • Raw totals as the headline.
  • Changing coverage without noting it.
  • No distinction between regressions and backlog.
  • Dashboards nobody reviews.

Testing checklist

Testing it with a screen reader

Trends cover automated findings only. Each quarter, pick the pages with the best trend lines and run a screen reader smoke test on them. If users still cannot complete key tasks, the trend is measuring the wrong things. Add manual findings to the same dashboard as a separate series so the gap between automated and real-world accessibility stays visible.

Leaders need three numbers, not a wall of charts: regressions introduced this period, median time to fix them and backlog density compared with last quarter. Add one sentence on the biggest systemic fix and one on the biggest remaining risk. Always include the coverage note (“60 page states, unchanged”) so nobody mistakes a coverage change for progress, and include the manual findings series to remind everyone what automation cannot see.

Storing results cheaply

You do not need a data warehouse to start. Upload each run’s JSON as a CI artifact, and run a nightly job that downloads the last few dozen artifacts, computes metrics and writes a small static HTML report or a CSV that a spreadsheet can chart. Move to a database only when queries become slow or you need cross-repository views. What matters is that the data is kept and looked at, not where it lives.

Setting targets

Set targets on behaviour, not totals: “no new critical regressions reach production” and “regressions fixed within five working days” are achievable and meaningful.

Key points

  • Normalise by scanned states; never headline raw totals.
  • Fingerprint violations to separate new regressions from backlog.
  • Measure time to fix, and group by component to find systemic causes.
  • Keep manual findings next to automated trends.

Design system notes

Tag violations with the design system component responsible where possible, so component owners see their share of the trend.

FAQ

Which metric matters most?

New regressions per release and time to fix.

How often should scans run?

On every main-branch build, with nightly full runs.

Should every violation block merges?

New serious and critical regressions should; backlog should not.

How do I handle axe upgrades?

Annotate the dashboard and re-baseline if rules change significantly.

Can I compare teams?

Use density and regression rates, not raw totals.

Do trends prove accessibility?

No; they cover automated checks only.

← Back to Automated Accessibility Testing Pipelines