A Singapore Government Agency Website

CLI options, environment variables and error codes

CLI options

Run the CLI with npm run cli -- <OPTIONS>. On Windows PowerShell, use npm run cli -- -- if options are not recognised. Refer to Run a scan for how to use them.

Required options

Option Long name Accepted values Default Description
-c --scanner sitemap (1), website (2), custom (3), intelligent (4), localfile (5) None Scan type. Required.
-u --url URL or file path None Website, sitemap or file to scan. Required.
-k --nameEmail "<NAME>:<EMAIL>" None Name and email address, separated by a colon. Required unless OOBEE_DISABLE_TELEMETRY is 1, true or yes.

Scan options

Option Long name Accepted values Default Constraints
-d --customDevice Desktop, Mobile or a device name Desktop Chrome Cannot be used with -w
-w --viewportWidth 320 to 1080 None Cannot be used with -d
-p --maxpages Positive integer 100 Not available for custom flow scans
-s --strategy same-domain, same-hostname, ignore ignore for sitemap scans, otherwise same-domain ignore is valid only for sitemap scans. Not available for local file scans.
-l --scanDuration Number of seconds, 0 or more 0 (no limit) None
-t --specifiedMaxConcurrency Positive integer 3 × processor cores, minimum 2, maximum 25 None
-i --fileTypes all, pdf-only, html-only html-only None
-f --safeMode yes, no no yes stops Oobee clicking buttons and links to find more links
-r --followRobots yes, no no yes makes the crawler follow robots.txt rules
-x --blacklistedPatternsFilename Path to a text file exclusions.txt One regular expression per line. Unsafe expressions are refused.
-y --ruleset default, disable-oobee, enable-wcag-aaa, disable-oobee,enable-wcag-aaa default default cannot be combined with another value
-j --customFlowLabel Text up to 80 characters None No forbidden folder-name characters. Applies to custom flow scans.

Browser and output options

Option Long name Accepted values Default Description
-b --browserToRun chromium (1), chrome (2), edge (3) Chrome on Windows and macOS, Chromium on Linux Browser to scan with. If Chrome is not found, macOS falls back to WebKit, Windows to Edge and Linux to Chromium.
-h --headless yes, no yes Run the browser in the background. Custom flow scans always show the browser.
-o --zip File name None Also write the results to this zip file
-e --exportDirectory Path to an existing folder results in the current folder Folder in which Oobee creates the scan folder. Oobee checks the path exactly as you type it. The CLI help says it is relative to your home directory, which is out of date.
-a --additional screenshots, none screenshots screenshots includes element screenshots in the report
-g --generateJsonFiles yes (y), no (n) no Keep the compressed JSON result files
-m --header "<HEADER> <VALUE>", pairs separated by a comma and a space None HTTP headers to send, for example credentials
-q --metadata JSON string {} Extra scan information for telemetry. An invalid string becomes {}.
-z --websiteTag Text None Tag that identifies the website in telemetry. Overrides OOBEE_TAGGED_WEBSITE.

The CLI exits with an error for any option that it does not recognise.

Devices

Pass Desktop, Mobile or a Playwright device name to -d. Desktop uses a 1280 × 720 viewport and Mobile uses the iPhone 11 viewport. In the interactive mode, the device choices are Desktop, Mobile, and Custom, which offers iPhone 11, Samsung Galaxy S9+ or a viewport width. Wrap a device name that contains brackets in single quotes.

The README lists the following device names. The list comes from Playwright’s device registry, and Playwright can add or remove devices between versions, so a name in this list may not exist in the Playwright version you use.

Family Names
Desktop Desktop Chrome, Desktop Edge, Desktop Firefox, Desktop Safari, and the HiDPI variants of Chrome, Edge and Firefox
iPhone iPhone 6, 6 Plus, 7, 7 Plus, 8, 8 Plus, SE, X, XR, 11, 11 Pro, 11 Pro Max, 12, 12 Pro, 12 Pro Max, 12 Mini, 13, 13 Pro, 13 Pro Max, 13 Mini
iPad iPad (gen 6), iPad (gen 7), iPad Mini, iPad Pro 11
Galaxy Galaxy Note 3, Galaxy Note II, Galaxy S III, Galaxy S5, Galaxy S8, Galaxy S9+, Galaxy Tab S4
Pixel Pixel 2, Pixel 2 XL, Pixel 3, Pixel 4, Pixel 4a (5G), Pixel 5
Nexus Nexus 4, 5, 5X, 6, 6P, 7, 10
Other Blackberry PlayBook, BlackBerry Z30, Kindle Fire HDX, LG Optimus L70, Microsoft Lumia 550, Microsoft Lumia 950, Nokia Lumia 520, Nokia N9, Moto G4

Add landscape to a device name for its landscape version, for example iPhone 13 landscape. The README list shows a landscape variant for every device except the desktop ones.

Environment variables

Set environment variables in the terminal before you run Oobee. Booleans accept 1, true or yes unless a row says otherwise.

Logging and behaviour

Variable Description Default
OOBEE_VERBOSE When set, log output goes to the console Off
OOBEE_LOGS_PATH Folder to write logs to None
OOBEE_VALIDATE_URL When set, Oobee checks the URL and exits without scanning Off
OOBEE_SLOWMO Slows the browser by this many milliseconds. Experimental. None
OOBEE_DISABLE_BROWSER_DOWNLOAD Stops file downloads in Chrome, Chromium and Edge. Experimental. Does not affect local file scans. Off
OOBEE_FAST_CRAWLER Listed in the README, but the engine source in version 0.11.19 does not read it, so it has no effect. false
OOBEE_IN_CONTAINER 1 forces container mode for runtimes that Oobee cannot detect. Set automatically when a container is detected. Automatic
OOBEE_USER_AGENT User-agent string for scans. If set, Oobee does not detect one. Detected, with HeadlessChrome replaced by Chrome
OOBEE_CHROME_DATA_DIR Folder to use as the Chrome data folder None
OOBEE_ALLOW_INSECURE_TLS Accepts invalid or self-signed certificates. Applies only when no credentials are sent. Off

Limits and timing

Variable Description Default
OOBEE_MAX_SCAN_MINUTES Upper limit on scan time in minutes. It can shorten -l but not lengthen it. None
OOBEE_MAX_IDLE_MINUTES Ends the scan and builds a report after this many minutes without a successful page scan. 0 turns it off. 0
OOBEE_CONSECUTIVE_MAX_RETRIES Ends the crawl after this many consecutive HTTP failures. 0 turns it off. 0
OOBEE_MAX_RATCHET_CYCLES Ends the crawl after this many concurrency halvings without full recovery. 0 turns it off. 0
OOBEE_RATE_RECOVERY_INTERVAL Successful pages needed before concurrency is raised again 5
OOBEE_MEM_PRESSURE_HIGH Memory use ratio above which concurrency is lowered 0.9
OOBEE_MEM_PRESSURE_LOW Memory use ratio below which concurrency is raised 0.75
OOBEE_LOAD_TIMEOUT_MS Wait for the page to load 30000
OOBEE_STABILITY_TIMEOUT_MS Wait for the page to stop changing 30000
OOBEE_QUIET_MS Period with no page changes that counts as settled 1500
OOBEE_MAX_MUTATIONS Page changes after which Oobee stops waiting 5000
OOBEE_ASSET_WAIT_MS Wait for page assets 5000
OOBEE_IDLE_TIMEOUT_MS Upper limit of each wait for the main thread to be idle 10000
OOBEE_PAGE_LOAD_BUDGET_MS Overall limit across all page-load waits. Keep it below the 90-second handler limit. 60000
OOBEE_IDLE_CALLBACKS Idle periods in a row that count as idle 3
OOBEE_AXE_RECHECK_HYDRATION_MS Delay before rechecking some violations that late page updates can change. 0 skips the delay. 1000
OOBEE_AXE_RECHECK_IDLE_TIMEOUT_MS Upper limit of the idle wait before the recheck. 0 turns it off. 4000
OOBEE_CLICK_DISCOVERY_MAX_MS Time limit on the click-discovery loop for each page 30000

Page capture

Variable Description Default
OOBEE_SAVE_DOM 1 or true saves each page’s DOM for desktop and mobile (iPhone 11 width) in the results folder Off
OOBEE_SAVE_PAGE_SCREENSHOT 1 or true saves full-page desktop and mobile screenshots in the results folder Off
OOBEE_SAVE_COMPUTED_STYLES Saves computed-style records for each viewport. Accepts the usual true and false values. Off
OOBEE_CAPTURE_MAX_ELEMENTS Most computed-style records for each viewport 100000
OOBEE_CAPTURE_MAX_STYLES_BYTES Approximate size limit of computed-style records for each viewport 100 MB
OOBEE_CAPTURE_MAX_DOM_BYTES Largest DOM saved for each viewport. Larger pages are skipped. 100 MB
OOBEE_HTML_MAX_BYTES Most bytes of HTML saved for each affected element 1024
OOBEE_PARENT_HTML_DEPTH Parent levels of HTML saved with each element. 0 turns it off. 0
OOBEE_PARENT_HTML_MAX_BYTES Most bytes of parent HTML Value of OOBEE_HTML_MAX_BYTES
OOBEE_PDF_MAX_DOWNLOAD_BYTES Largest PDF Oobee downloads 100 MB
OOBEE_PDF_MAX_SCREENSHOTS Most PDF screenshots for each scan. 0 for no limit. 500
OOBEE_SCANHTML_MAX_BYTES Largest HTML string scanHTML accepts. 0 for no limit. 50 MB
OOBEE_SCANHTML_AXE_TIMEOUT_MS Time limit for one scanHTML run. 0 for no limit. 300000

Telemetry and tagging

Variable Description Default
OOBEE_DISABLE_TELEMETRY 1, true or yes stops all telemetry. Name, email and entry URL are never sent, and -k becomes optional. Off
OOBEE_TELEMETRY_ALLOW_PII Allows name, email and a user ID to be attached to error-monitoring events. Without it, they are left out. Off
OOBEE_TAGGED_WEBSITE Tag that identifies the website in telemetry. -z takes precedence. None
OOBEE_SCAN_METADATA Replaces the entryUrl tag sent to telemetry None
OOBEE_SCAN_PRODUCT Adds a scanProduct tag to telemetry events None

Security

Variable Description Default
OOBEE_SSRF_PROTECTION Restricts scanCustomFlow to public http and https targets. Needed when URLs come from untrusted users. Off
OOBEE_ALLOW_INTERNAL_TARGETS 1 allows a public site’s links to point at private-range addresses. By default, Oobee refuses these when the start URL is public. Off
OOBEE_SCOPE_HEADERS_TO_ORIGIN 1 sends operator headers only to the exact origin of the start URL. The default sends them to the same registrable domain. Off
OOBEE_UNSCOPED_OPERATOR_HEADERS 1 sends operator headers to every request Off

Proxy and network

Variable Description Default
HTTP_PROXY, HTTPS_PROXY, ALL_PROXY Proxy server for HTTP, HTTPS or all requests None
NO_PROXY Domains that skip the proxy, separated by commas None
INCLUDE_PROXY Domains that use the proxy, separated by commas or semicolons. Cannot be combined with NO_PROXY. None
PROXY_USERNAME, PROXY_PASSWORD Proxy credentials. HTTP_PROXY_USERNAME and HTTP_PROXY_PASSWORD are also read. None
CF_WORKER_PROXY Experimental. URL of a Cloudflare Worker proxy. Overrides the proxy variables above. None
CF_WORKER_PROXY_AUTH_TOKEN Sent as the Authorization header to the Worker None
CF_WORKER_PROXY_PORT Local port of the Worker tunnel 8877
CF_WORKER_PROXY_FORCE_TUNNEL_HOSTS Host patterns always sent through the Worker None
FAMILY_DNS 1, true or yes turns on family-filtering DNS. A blank value counts as unset. Off
CF_FAMILY_DNS Deprecated alias of FAMILY_DNS. Ignored when FAMILY_DNS is set. Off

Refer to Configure a proxy and Enable Safe Browsing and family DNS for how to use them.

Safe Browsing

Variable Description Default
GOOGLE_SAFE_BROWSING Any value turns on Safe Browsing. Needs Google Chrome. Off
SB_PROFILE_DIR Folder for the prepared Safe Browsing profile ~/.oobee/safe-browsing-profile
SB_DB_TIMEOUT_MS Time to wait for the database download 300000
SB_PREPOPULATED_ZIP Path to a pre-populated database zip file None
SB_PREPOPULATED_DIR Path to a pre-populated database folder None
SB_PREPOPULATED_SHA256 SHA-256 digest that a pre-populated zip must match. A zip that does not match is rejected. None

Amazon S3 upload

Variable Description Default
S3_BUCKET_NAME, OOBEE_SCAN_ID, OOBEE_USER_ID, OOBEE_USER_EMAIL All four must be set to turn on the upload None
AWS_REGION Region of the bucket ap-southeast-1
OOBEE_MESSAGE_ID, OOBEE_AMPLITUDE_USER_ID, OOBEE_DEVICE_ID, OOBEE_ORG_ID, OOBEE_USER_ROLE Optional file metadata None
CLOUDFRONT_BASE_URL Base of the file URLs in the log Bucket URL

Refer to Upload results to Amazon S3.

Variables that Oobee sets itself

Do not set these. Oobee sets them, and changing them can change behaviour unexpectedly.

Variable Set by Oobee to
CRAWLEE_STORAGE_DIR The Crawlee dataset folder
CRAWLEE_LOG_LEVEL ERROR
CRAWLEE_HEADLESS 1 for headless, 0 for headed

URL validation error codes

Oobee checks the start URL or file before it scans. A failed check ends the run with the code below as the exit code and prints the message.

Code Name Message What to check
0 success None No action needed
11 invalidUrl “Invalid URL. Please check and try again.” The URL starts with http:// or https:// and has no typos
12 cannotBeResolved “URL cannot be accessed. Please verify whether the website exists.” The domain name is correct, DNS resolves and the site is public
13 errorStatusReceived “Provided URL cannot be accessed. Server responded with code” followed by the status The server’s HTTP status
14 systemError “Something went wrong when verifying the URL. Please try again in a few minutes. If this issue persists, please contact the Oobee team.” Retry, and check your internet connection
15 notASitemap “Invalid sitemap URL format. Please enter a valid sitemap URL ending with .XML or .TXT e.g. https://www.example.com/sitemap.xml." The URL points to a valid sitemap
16 unauthorised “Login required. Please enter your credentials and try again.” The site needs credentials. Refer to Scan pages behind a login.
17 browserError “Incompatible browser. Please ensure you are using Chrome or Edge browser.” A supported browser is installed
18 sslProtocolError “SSL certificate error. Please check the SSL configuration of your website and try again.” The certificate is valid and not expired, and TLS settings match
19 notALocalFile “Uploaded file format is incorrect. Please upload a HTML, PDF, XML or TXT file.” The file is an HTML, PDF, XML or TXT file
20 notAPdf “URL/file format is incorrect. Please upload a PDF file.” The file is a PDF, when -i pdf-only is set
21 notASupportedDocument “Uploaded file format is incorrect. Please upload a HTML, PDF, XML or TXT file.” The file type is allowed by -i
22 connectionRefused “Connection refused. Please try again in a few minutes. If this issue persists, please contact the Oobee team.” The server is running and no firewall blocks it
23 timedOut “Request timed out. Please try again in a few minutes. If this issue persists, please contact the Oobee team.” Your connection and the server load
24 blockedByClient “Something went wrong when verifying the URL. If this issue persists, please contact the Oobee team.” The browser reported that the page was blocked by the client or by the response, for example by Safe Browsing

Files Oobee reads and writes

File or folder Location Purpose
results/ The folder where you run Oobee, unless -e is set Scan results, one folder for each scan
exclusions.txt The Oobee folder Default list of URL patterns to exclude. Refer to Scan a user flow with custom flow.
userData.txt Windows: %APPDATA%\Oobee. macOS: ~/Library/Application Support/Oobee. Linux: ~/.config/oobee. Cached name, email and a user ID
<UUID>.txt OOBEE_LOGS_PATH if set. Otherwise %APPDATA%\Oobee on Windows, ~/Library/Application Support/Oobee on macOS and the current folder on Linux. Log file for a run

To change the cached name and email address, delete userData.txt. Oobee asks for them again on the next interactive run.

Chat with me!
Bot Launcher