How Oobee scans
Overview of an Oobee scan
Oobee is an accessibility scanner that visits web pages in a real browser, runs automated checks against each page and builds a report of the issues it finds. The browser matters: Oobee checks the page after the browser has run its scripts, so it tests what a visitor sees and not the raw HTML. The checks come from the axe-core engine plus Oobee’s own rules.
An Oobee scan has four stages:
| Stage | What happens |
|---|---|
| Input | The command line, the interactive prompts or the npm module passes a scan type, a start point and options to the orchestrator |
| Discovery | A crawler finds the pages to scan from a sitemap, from links, or both |
| Page scan | Each page opens in a browser, settles, and is checked. The result is saved as one record per page. |
| Reporting | Oobee reads all the records, groups the issues and writes the HTML, PDF, CSV and optional JSON reports |
The technology behind these stages is Crawlee for crawling, Playwright for browser control, axe-core for checks, VeraPDF for PDF files and Node.js to run it.
How the parts fit together
The following diagram shows the path of a scan from your input to the reports. The scan type chooses the crawler. Every crawler uses the same browser-based page scan and writes to the same store of per-page results, and the reporting stage reads that store.
The orchestrator receives the scan from the command line, the interactive prompts or the npm module, and routes it by scan type. Website, sitemap, local file and custom flow scans each have their own crawler. The intelligent crawler runs the sitemap crawler first and then the domain crawler. All crawlers send each page to the same browser-based page scan, which writes one result record per page to a store. The reporting stage reads every record from that store and writes the reports.
Scan types and how they find pages
The scan type decides how Oobee discovers pages. This table summarises each type. Refer to Run a scan for the commands.
| Scan type | How it finds pages | Order of pages |
|---|---|---|
| Website | Starts at one page and follows the links it finds, then visits the scanned pages again and clicks interactive elements to find more links | Not fixed |
| Sitemap | Reads the URLs in a sitemap file | Mostly the order in the file |
| Intelligent | Finds the sitemap from robots.txt or common sitemap paths, scans it, follows links to pages the sitemap misses, and falls back to a website scan if no sitemap exists |
Sitemap order, then not fixed |
| Local file | Scans one HTML or PDF file, or the pages listed in a local sitemap file | One file, or the file’s order |
| Custom flow | Scans the pages you open and choose in a visible browser | The order you choose |
Intelligent scans give the most predictable coverage on a live site. The sitemap ensures that pages you know about are scanned in a consistent order, and link discovery catches pages the sitemap does not list. For an intelligent scan, Oobee skips the click-discovery pass, because sitemap reading and link extraction already cover it.
Scan strategy
The strategy option decides which discovered links Oobee follows. It filters only links found while crawling. It does not change which pages a sitemap lists.
| Strategy | Follows | Default for |
|---|---|---|
same-domain |
Links on the same registered domain, including subdomains | Website, intelligent and custom flow scans |
same-hostname |
Links on the exact same host name, after removing www. |
None |
ignore |
Every URL, with no filter | Sitemap scans, and valid only for them |
For example, when you scan https://www.acme.gov.sg/services/home, a link to https://employer.acme.gov.sg/dashboard is followed under same-domain and skipped under same-hostname. A link to https://www.other.gov.sg/resources is skipped under both. Use same-hostname when following subdomains would spend the page budget on unrelated content.
Page budget and time limit
The page budget, set with -p, is the number of pages Oobee scans successfully. It counts successful scans only. A page that fails three times and then succeeds counts once, and a page that fails for good does not use up the budget. In an intelligent scan the sitemap and link phases share one budget. The time limit, set with -l, stops all phases when it runs out.
A budget works well as a sample because most accessibility issues repeat across page templates. Scanning a subset of a site often finds the same patterns as scanning all of it. CRAWL.md in the repository suggests the following starting points, and they are guidance and not limits:
| Site size | Suggested budget |
|---|---|
| Under 100 pages | Equal to or above the page count |
| 100 to 1000 pages | 500 to 1000 pages |
| Over 1000 pages | 2000 to 5000 pages |
Why some pages are not scanned
Oobee records every page it could not scan, with a reason, and lists it in the report. The reasons come from what happens when the page loads.
| Reason | What it means |
|---|---|
| Not an HTML document | The URL is an image, spreadsheet, video, download or a PDF while PDF scanning is off. These appear as Unsupported Documents. |
| Blocked by the server | A firewall or rate limiter returned an error, most often 403 Forbidden |
| Needs a login Oobee does not have | The page needs credentials that were not provided |
| Too slow | The page did not start loading within 30 seconds |
| Budget used | The page or time limit was reached first |
| Redirected out of scope | The page redirected to a domain that the strategy excludes |
| Excluded | The URL matched an exclusion pattern or a file extension that Oobee never scans |
Before it opens the browser, Oobee skips URLs with known non-HTML file extensions and links that are not web pages, such as mailto: and tel:. When robots.txt following is on, it also skips paths that robots.txt disallows.
How Oobee handles errors and retries
A page goes through a fixed sequence. Navigation has a 30-second timeout. If it succeeds, Oobee waits for the page to load and settle, then runs the checks. The whole handling of one page has a 90-second limit. If any step fails, Crawlee retries the page up to three times. Only after the last retry fails does Oobee classify the page and record it, so a page that succeeds on a retry is never recorded as an error.
| Outcome after retries | Recorded as |
|---|---|
| A download starts instead of a page | Unsupported Document |
| First 403 or 429 response | Put back at the end of the queue for one more try |
| Second 403 or 429 response | Not scanned, with the status code |
| Any other failure | Web Crawler Errored |
| Chrome Safe Browsing blocks the page | Blocked by Safe Browsing |
How Oobee adapts to rate limiting and load
Oobee scans up to 25 pages at once by default, or three times the number of processor cores if that is lower, with a minimum of two. A rate controller lowers this number when conditions get worse and raises it again when they recover. It watches three signals.
| Signal | Response |
|---|---|
| A 403 or 429 from the site | Halve the number of simultaneous pages. After five successful pages, double it again, up to the starting value. |
| Pages whose main thread never goes idle, which indicates a starved processor | Reduce by a quarter, and raise again when pages settle |
| Browser crashes or high memory use | Halve after a crash, reduce by a quarter after sustained high memory use, and raise by one when memory has headroom |
Two optional limits stop a scan that the site keeps blocking: OOBEE_CONSECUTIVE_MAX_RETRIES and OOBEE_MAX_RATCHET_CYCLES. Both are off by default, so a scan survives a short burst of 403 errors. OOBEE_MAX_IDLE_MINUTES ends a scan that makes no successful scan for that long and builds a report from the pages already scanned. Refer to CLI options, environment variables and error codes for all of them.
Browsers and sessions
Oobee retires a browser after 500 pages and starts a new one. Each browser has its own temporary profile folder, cloned from the original so that cookies stay fresh. The original profile is never changed by a running browser, and Oobee deletes the temporary folders during and after the scan. A site that tracks sessions on the server may treat the scanner as a new visitor after a browser is replaced. Oobee also caps each browser’s disk cache at 10 MB to keep storage use down.
What affects the accuracy of results
| Factor | Effect on results |
|---|---|
| Dynamic content | A page with heavy JavaScript can give slightly different results between runs, because content loads at different times. Oobee waits for the page to settle, but some elements can appear after the checks. |
| Slow machines | On a slow or throttled machine the browser runs slowly, so a page can look settled before it has rendered. The scan then runs on an incomplete page and can miss issues or flag placeholder content. Pages that time out are retried three times and then recorded as errors. |
| Rate limiting | Pages that a site keeps blocking can be lost. A larger page budget and a longer time limit help. |
| Login state | Oobee sees a site as a signed-out visitor unless you provide credentials, headers or cookies. |
| Link discovery in website scans | Pages load at the same time, so the order in which links are found changes between runs. A page found in one run may be missed in another if the limit is reached first. |
| Redirects during a scan | If a page redirects after it loads, the check may run on the redirect target. |
A sitemap scan finds the same pages on every run, because its input is fixed. A website scan can find different pages on different runs.
Related content
- Run a scan — scan types and options.
- Read and export scan reports — what the report shows about scanned and skipped pages.
- Scan rules and severity — the rules behind the checks.
- CLI options, environment variables and error codes — every option and limit.
- Troubleshooting — pages that are skipped or scans that stop early.
