Gate a CI build on accessibility thresholds

In this tutorial, you build a script that scans a page with Oobee and fails the build when the number of issues passes limits you set. By the end, you have a Playwright script that exits with a non-zero code on a breach, and a GitLab CI/CD job that runs it and keeps the report. This tutorial is for developers who have run an Oobee scan before and want to enforce accessibility limits in a pipeline.

Prerequisites

• Oobee installed and a first scan completed. Refer to Get started with Oobee.
• Node.js 20 or later and a Node.js project with a package.json file.
• The URL of a page that the CI runner can reach and that you are allowed to scan.
• A GitLab project with CI/CD runners that can run Docker images.
• Familiarity with the instance and thresholds option. Refer to npm API.

Step 1: Install Oobee and Playwright

In your project folder, install the packages and the Playwright browser:

npm install --save-dev @govtechsg/oobee playwright
npx playwright install chromiumCopy to clipboardErrorCopied

If the import in the next step fails because the package has no dist folder, build the package:

cd node_modules/@govtechsg/oobee
npm install
npm run build
cd ../../..Copy to clipboardErrorCopied

Step 2: Write the scan script

Create a11y-gate.mjs. The script creates an Oobee instance with thresholds, scans one page, and ends with a report. When a threshold is passed, testThresholds ends the Oobee instance and throws, and the script sets a failing exit code.

import { chromium } from "playwright";
import oobeeA11yInit from "@govtechsg/oobee";

const pageUrl = process.env.A11Y_PAGE_URL;

const oobeeA11y = await oobeeA11yInit({
  entryUrl: pageUrl,
  testLabel: "CI accessibility gate",
  name: process.env.OOBEE_NAME,
  email: process.env.OOBEE_EMAIL,
  includeScreenshots: false,
  thresholds: { mustFix: 0, goodToFix: 25 },
});

const browser = await chromium.launch({ headless: true });

try {
  const page = await browser.newPage();
  await page.goto(pageUrl);
  await page.evaluate(oobeeA11y.getAxeScript());
  await page.evaluate(oobeeA11y.getOobeeFunctions());

  const scanResults = await page.evaluate(async () => await runA11yScan([]));
  const counts = await oobeeA11y.pushScanResults(scanResults, undefined, undefined, page);
  console.log(`Must Fix: ${counts.mustFix}, Good to Fix: ${counts.goodToFix}`);

  oobeeA11y.testThresholds();
  await oobeeA11y.terminate();
} catch (error) {
  console.error(error.message);
  process.exitCode = 1;
} finally {
  await browser.close();
}Copy to clipboardErrorCopied

The thresholds count issue occurrences, not rules. mustFix: 0 fails the build on any Must Fix issue. goodToFix: 25 allows up to 25 Good to Fix occurrences. Oobee fails a threshold only when the count is greater than the limit.

Step 3: Run the script and trigger a failure

Set the variables and run the script against your page:

export A11Y_PAGE_URL="<PAGE_URL>"
export OOBEE_NAME="<YOUR_NAME>"
export OOBEE_EMAIL="<YOUR_EMAIL>"
node a11y-gate.mjs
echo "Exit code: $?"Copy to clipboardErrorCopied

The script prints the two counts. If both are within the limits, the exit code is 0 and Oobee writes the report to a folder under results. To see a failure, set mustFix to a number below the count the script printed, run it again, and check the exit code. The script prints a message in this form and the exit code is 1:

Exceeded thresholds:
mustFix occurrences found: <COUNT> > <LIMIT>Copy to clipboardErrorCopied

Restore your real limits before you continue.

Step 4: Add the script to a GitLab CI/CD job

Run the script in a job that uses a Playwright image, so the browser and its system libraries are present. The Oobee Dockerfile uses this image as its base.

a11y-gate:
  stage: test
  image: mcr.microsoft.com/playwright:v1.61.1-noble
  variables:
    A11Y_PAGE_URL: "<PAGE_URL>"
    OOBEE_NAME: "<YOUR_NAME>"
    OOBEE_EMAIL: "<YOUR_EMAIL>"
  script:
    - npm ci
    - node a11y-gate.mjsCopy to clipboardErrorCopied

A non-zero exit code from node a11y-gate.mjs fails the job, and GitLab then blocks the pipeline stages after it. If npm ci does not build Oobee, add the build commands from Step 1 before the scan.

Step 5: Keep the report as a job artefact

Add an artifacts block that keeps the results, including when the job fails, so you can open the report and see what broke.

  artifacts:
    when: always
    paths:
      - results/
    expire_in: 1 weekCopy to clipboardErrorCopied

After a run, download the job artefacts and open report.html in the newest folder under results. Refer to Read and export scan reports.

What you have built

You now have a script that scans a page with Oobee, compares the counts with limits you set, and exits with an error when a limit is passed. You also have a GitLab CI/CD job that runs the script and keeps the report. You used oobeeA11yInit, getAxeScript, getOobeeFunctions, pushScanResults, testThresholds and terminate.

Common issues

Next steps

• Integrate Oobee with Playwright — scan several pages and parts of pages in one run.
• Run Oobee in CI/CD — scan a whole site with the CLI.
• npm API — every option, including ruleset for AAA rules.