These guidelines apply to changes to the Oobee engine source in src/. They are for contributors who change crawlers, browser handling, report generation or the report templates. Each rule says what to do and why, because the engine has edge cases that tests do not always catch. Rules use “must” for requirements and “should” for recommended practice.
A new browser context that fetches remote content in headless mode must pass userAgent: process.env.OOBEE_USER_AGENT.
The default user agent contains HeadlessChrome. Firewalls block requests that use it.
Do not set Authorization in extraHTTPHeaders on a context. Use splitAuthHeaders() and addAuthRouteHandler() or preNavigationHooks.
Playwright sends extraHTTPHeaders to every request, including other origins such as content delivery networks, and that causes cross-origin preflight failures.
Do not change the shared extraHTTPHeaders object. Copy it first: const localHeaders = { ...extraHTTPHeaders }.
A change makes the object non-empty, which turns on header rewriting and a Playwright performance warning for every unauthenticated scan.
In preNavigationHooks, change gotoOptions in place. Do not assign a new object to the parameter.
Assigning rebinds only the local variable, so Crawlee never sees the change.
Use domcontentloaded as the navigation wait, not networkidle.
Sites with WebSockets, analytics polling or beacons never go quiet, so networkidle can hang.
Headed mode must use launch() with newContext(). It must not use launchPersistentContext with a custom userDataDir.
The persistent form causes a “Browser window not found” crash.
Use isRunningInContainer() to detect containers. Do not check for /.dockerenv.
Fargate has no /.dockerenv.
A custom flow must check the entry URL’s scheme and host, and set OOBEE_SSRF_PROTECTION=1 when URLs come from untrusted users.
Without it, the scanner can reach internal hosts and local files and put what it finds in a report.
maxRequestsPerCrawl counts successful scans only. Call claimSlot() synchronously immediately before pushing to urlsCrawled.scanned.
A page can be discarded part-way through its handler, for example after a redirect or robots.txt block. Claiming earlier over-counts.
Call abort() only after the last slot is claimed. Never abort from an early check at the top of a handler.
Aborting early kills pages that other handlers are still scanning, and the scan finishes short of its limit.
Record an error only in failedRequestHandler, never in the requestHandler catch block.
Crawlee retries a page up to three times. Recording earlier creates duplicate errors for pages that succeed on a retry.
Handle 403 and 429 in requestHandler, with sessionPoolOptions.blockedStatusCodes: [] and retryOnBlocked: false.
Crawlee’s defaults retire the session and browser before the handler runs, so each blocked URL costs about eight navigations and eight browser launches.
Do not set a minConcurrency floor.
A floor of 10 kept more than 20 renderers alive on 2 processors when two scans shared a task, and that starved page hydration.
Treat scanDuration of 0 as no limit.
Code that computes remaining time otherwise reads 0 as no time left.
Skip strategy checks when userUrl is a file path.
new URL() throws on a file path.
Both crawlers must pass the same parameters to runAxeScript(), including ruleset. Add any new parameter to every crawler through combine.ts.
Scan types must give the same findings for the same page.
Both crawlers must use an identical DOM observer in postNavigationHooks, including the observer.observe(root, { childList: true, subtree: true }) call.
Without the call, the hook falls back to a fixed 5-second timeout and the page may be in a different state when it is scanned.
Wrap page.evaluate() in postNavigationHooks in try/catch.
Pages can be destroyed during the observer, and the error would reach Crawlee’s retry logic differently in each crawler.
Keep the report’s embedded data to rule summaries with pagesAffectedCount. Do not embed full item arrays.
A browser cannot run JSON.parse() on strings of 700 MB or more.
Wait for the drain event when you write chunked Base64 data to the HTML output stream.
Without it, payloads of 57 MB or more are truncated silently.
Delete the dataset folder and the temporary item store before zipping. Zip as the last step of generateArtifacts().
Zipping earlier includes temporary files.
Serialise writes to each per-rule JSONL file, and remove literal \n and \r from text after JSON.stringify().
Interleaved writes or stray newlines corrupt a line, and the reader skips corrupt lines.
Capture document.title at the start of runAxeScript(). Do not open a page to read it later.
Pages can close during scanning, and opening another page leaks pages.
Give top-level names in EJS partials a partial-specific name, for example SAFE_SCANNED_HREF_SCHEMES.
All partials share one script scope. A repeated top-level name stops that whole script block, and a section of the report renders empty without an error.
Test template changes by rendering the whole report and compiling all its scripts together.
tsc does not check EJS, and each file is valid alone.
The following examples show two rules in code. Each shows the poor form first, then the good form.
// Poor: assigns a new object, so Crawlee never sees the change
preNavigationHooks: [
(crawlingContext, gotoOptions) => {
gotoOptions = { ...gotoOptions, waitUntil: 'domcontentloaded' };
},
],
// Good: changes the object in place
preNavigationHooks: [
(crawlingContext, gotoOptions) => {
gotoOptions.waitUntil = 'domcontentloaded';
gotoOptions.timeout = 30000;
},
],
// Poor: compares host names directly, so www.example.com and example.com differ
if (new URL(link).hostname === new URL(entryUrl).hostname) { /* follow */ }
// Good: removes www. before comparing the two host names
if (isSameHostname(new URL(link).hostname, new URL(entryUrl).hostname)) { /* follow */ }
