Oobee architecture

Who this page is for

This page describes how the Oobee engine is built. It is for people who change the engine’s source code. To use Oobee, refer to the user guide: How Oobee scans explains scanning from a user’s point of view.

Engine overview

Oobee is a TypeScript project that runs on Node.js. It crawls web pages with Crawlee’s PlaywrightCrawler, checks each page with axe-core and Oobee’s own checks, and writes HTML, PDF, CSV and JSON reports.

The flow is:

User input (CLI, interactive prompts or npm module)
  -> orchestrator (src/combine.ts), which routes by scan type
  -> crawler (domain, sitemap, intelligent, local file or custom flow)
  -> page handler (axe-core injection and Oobee checks)
  -> per-page records in a Crawlee dataset
  -> artefact generation (src/mergeAxeResults.ts)
  -> reports (HTML, PDF, CSV, JSON)

Entry points

Entry	File	Purpose
CLI	`src/cli.ts`	yargs-based command line. Calls the orchestrator.
Interactive CLI	`src/index.ts`	Prompts with Inquirer. Calls the orchestrator.
npm module	`src/npmIndex.ts`	`init()`, `scanPage`, `scanHTML` and `scanCustomFlow` for other programs
Orchestrator	`src/combine.ts`	Routes by scan type, manages the scan’s life cycle and calls artefact generation

Scan types and crawlers

Scan type	File	Behaviour
`Website`	`src/crawlers/crawlDomain.ts`	Crawls a domain and discovers links from pages
`Sitemap`	`src/crawlers/crawlSitemap.ts`	Fetches URLs from sitemap XML
`Intelligent`	`src/crawlers/crawlIntelligentSitemap.ts`	Finds a sitemap through `robots.txt`, crawls it, then adds a domain crawl
`LocalFile`	`src/crawlers/crawlLocalFile.ts`	Scans local HTML or PDF files through `file://`
`Custom`	`src/crawlers/runCustom.ts`	A user-driven flow in a visible browser. `scanCustomFlow.ts` wraps it for the npm module.

All crawlers use PlaywrightCrawler with Crawlee’s own request limit turned off (maxRequestsPerCrawl: Infinity). They stop manually when the number of successful scans reaches the page limit. They also set useFingerprints: false and, in the domain and sitemap crawlers, retryOnBlocked: false.

Key source files

Area	File	Contents
Constants	`src/constants/constants.ts`	Enums (`ScannerTypes`, `BrowserTypes`, `FileTypes`, `RuleFlags`), browser data folder paths, sitemap path list, WCAG mappings and shared mutable state
Common helpers	`src/constants/common.ts`	URL validation (`checkUrl`), browser launch options (`getPlaywrightLaunchOptions`), sitemap parsing, `robots.txt` handling, browser choice (`getBrowserToRun`) and user-agent set-up (`initModifiedUserAgent`)
Rate control	`src/crawlers/crawlRateController.ts`	Adaptive concurrency and the optional circuit breakers
Page handling	`src/crawlers/commonCrawlerFunc.ts`	Page hooks, axe-core injection, result filtering
Artefacts	`src/mergeAxeResults.ts` and `src/mergeAxeResults/`	`generateArtifacts()`, JSON, CSV and sitemap writers, item store
Report templates	`src/static/ejs/`	EJS templates for the HTML report and the PDF summary
Proxy	`src/proxyService.ts`, `src/cfProxyWorker.ts`	Proxy detection, the Cloudflare Worker tunnel and family DNS
Safe Browsing	`src/safeBrowsingProfile.ts`	Database warm-up and injection into scan profiles
S3 upload	`src/services/s3Uploader.ts`	Upload of the results folder

Report generation

generateArtifacts() in src/mergeAxeResults.ts reads the Crawlee dataset, builds an allIssues object and writes all outputs. Each page’s results are stored as a numbered JSON file in <token>/datasets/default/. Violation items are written to per-rule JSONL files on disk, so the process does not hold a whole large scan in memory. Only rule-level summaries are embedded in report.html. The sub-modules in src/mergeAxeResults/ write the JSON and Base64 files, the CSV files and the sitemap.

The order of the final steps matters: Oobee deletes the dataset folder and the temporary item store before it zips the results, and zipping is the last step.

Browser handling

getBrowserToRun() in src/constants/common.ts chooses the browser.

Platform	Default	Fallback
macOS	Chrome	WebKit
Windows	Chrome	Edge, then an error
Linux and Docker	Chrome if present	Playwright’s bundled Chromium

Requesting chromium uses Playwright’s bundled Chromium with no channel. getPlaywrightLaunchOptions() builds the launch configuration:

Headless mode comes from CRAWLEE_HEADLESS.
Container detection adds --disable-gpu, --disable-dev-shm-usage and --no-zygote.
Proxy settings come from getProxyInfo().
--mute-audio is on by default, and off for custom flow.

initModifiedUserAgent() replaces HeadlessChrome with Chrome in the user agent and stores it in OOBEE_USER_AGENT. Call it before any browser context that talks to remote servers in headless mode, or firewalls may reject the requests. A user agent that is already set in OOBEE_USER_AGENT is kept.

Shared mutable state

The default export of constants.ts holds state for the current run.

Field	Holds
`constants.launcher`	The Playwright browser type (Chromium or WebKit)
`constants.robotsTxtUrls`	Parsed `robots.txt` rules
`constants.sitemapFetchedLinks`	Sitemap fetch details. Reset to `null` at the start of each scan.
`constants.userDataDirectory`	The current browser profile folder
`constants.randomToken`	The current scan token
`constants.resources`	Active crawlers, contexts and browsers, for clean-up

Platform differences

Platform	Notes
Docker or Linux	Detected by `isRunningInContainer()`, which covers Docker, Podman, Kubernetes, ECS and Fargate, Azure and Cloud Run, or `OOBEE_IN_CONTAINER=1`. Always headless. No Chrome profile to clone. Default concurrency is 3 × available processors, between 2 and 25.
macOS	Uses system Chrome if present. Typically headed. Logs are in `~/Library/Application Support/Oobee/`.
Windows	Uses system Chrome, then Edge. File locks need longer clean-up delays: 5 seconds against 3 seconds elsewhere. `CRAWLEE_SYSTEM_INFO_V2=1` is needed because `wmic` is deprecated.

Use isRunningInContainer() and not a check for /.dockerenv, because Fargate has no such file.

Build and test Oobee — set up, build and test the engine.
Contributor guidelines — rules and pitfalls to keep in mind when changing the engine.
How Oobee scans — the user’s view of scanning.

Contributing

Build and test Oobee