Skip to content

About

A CI-oriented tool that compares accessibility scans between two builds and blocks only newly introduced, evidence-backed regressions.

Topics

Resources

Stars

1 star

Watchers

0 watching

Forks

Repository files navigation

Web Accessibility Regression Guard

CI

A CI-oriented tool that compares accessibility scans between two builds. It blocks only newly introduced regressions with evidence from the scan.

Problem

Developers add accessibility issues by accident. Running an audit on every commit generates too much noise. Teams need a tool that compares the current build against a known baseline.

The tool must flag only regressions. It must not fail on pre-existing issues. It must provide structured output for CI systems.

Design

The tool stores scan results as JSON baseline files. Each page scan records violations, passes, and incomplete checks from axe-core.

The diff engine compares nodes by URL, rule ID, and CSS selector. A node present only in the current scan is a regression. A node present only in the baseline is a fixed issue. An entirely new violation rule marks all its nodes as new violations.

GitHub Actions workflow commands annotate pull requests. Each regression produces a warning or error annotation. A clean diff produces a notice.

The scanner uses Playwright to load pages and axe-core to audit them. It supports custom viewports, timeouts, WCAG tag filters, and rule exclusion lists.

Setup

git clone https://github.com/DanielCuevas1208/web-a11y-regression-guard.git
cd web-a11y-regression-guard
npm install
npm run build

Install Playwright browsers once if you plan to run scans:

npx playwright install chromium

Usage

Scan a set of URLs

Write a script that uses the scanner API. The method requires a Playwright Page object.

import { chromium } from "playwright";
import { scan } from "web-a11y-regression-guard";

const browser = await chromium.launch();
const page = await browser.newPage();

const report = await scan(page, {
  urls: ["https://example.com"],
  tags: ["wcag2a", "wcag2aa"],
});

await browser.close();

Save the report as your baseline.

Diff two scan reports

npx a11y-guard diff --baseline fixtures/scans/baseline.json --current fixtures/scans/current-with-regressions.json

Output:

FAIL: 3 regression(s) found across 2 violation rule(s).
  [SERIOUS] https://example.com/page-a - color-contrast: Elements must meet...
        Selector: .intro-text
  [CRITICAL] https://example.com/page-a - label: Form elements must have labels
        Selector: #signup-email
  [CRITICAL] https://example.com/page-a - button-name: Buttons must have...
        Selector: .close-btn

Generate GitHub annotations

npx a11y-guard diff \
  --baseline baseline.json \
  --current current.json \
  --annotate \
  --error-on-regression

The --annotate flag prints workflow commands to stdout. Pipe the output to the CI log for inline PR annotations.

Options

Option Description
--baseline Path to the baseline scan JSON file
--current Path to the current scan JSON file
--annotate Print GitHub Actions workflow commands
--error-on-regression Exit with code 1 when regressions exist
--title Title for the annotation group
--file Source file path for annotation context

Programmatic API

import { loadBaseline, diff, annotate } from "web-a11y-regression-guard";

const baseline = loadBaseline("baseline.json");
const current = loadBaseline("current.json");
const result = diff(baseline, current);

console.log(annotate(result, { mode: "error" }));

Architecture

src/
  types.ts         Shared type definitions
  baseline.ts      Load, save, and validate scan reports
  diff.ts          Compare two reports and find regressions
  annotations.ts   Format results as GitHub workflow commands
  scanner.ts       Playwright + axe-core integration
  cli.ts           Commander-based CLI entry point
  index.ts         Public API barrel export
fixtures/
  pages/           Sample HTML pages for testing
  scans/           Pre-recorded scan reports for testing
tests/
  baseline.test.ts   Baseline I/O and validation tests
  diff.test.ts       Regression detection tests
  annotations.test.ts  GitHub command formatting tests

Tests

The project uses fixture-driven tests. Each test loads a scan report from fixtures/scans/ and verifies expected behavior.

npm test

The test suite includes 38 deterministic tests. It covers baseline loading, diff computation, and annotation formatting.

Sample fixtures

Fixture Purpose
baseline.json Initial scan with two violations on two pages
current-clean.json Same as baseline, no new issues
current-with-regressions.json Three extra violation nodes added
current-shifted.json Violations moved between pages
pages/accessible.html WCAG-conformant HTML form
pages/inaccessible.html Page with common violations

Limitations

The tool compares exact scan reports. It does not tolerate URL changes or selector drift between builds. Use stable URLs and stable selectors.

It depends on axe-core for detection. Axe-core finds approximately 30% to 50% of WCAG issues. Manual review is still necessary.

The tool does not moderate between two concurrent branches. It requires a single baseline and a single current report.

It does not scan SPAs automatically. Write a script that navigates through each route before calling scan().

License

MIT

About

A CI-oriented tool that compares accessibility scans between two builds and blocks only newly introduced, evidence-backed regressions.

Topics

Resources

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages