Upload results to Amazon S3

What this guide covers

Oobee can upload the whole results folder to an Amazon Simple Storage Service (S3) bucket when a scan finishes. Each file is tagged with the scan ID, user ID and email you supply, so a downstream system, such as an S3 event trigger, can pick the files up and process them. This guide shows how to turn the upload on and where the files go. It does not cover creating the bucket or the downstream processing.

Prerequisites

An S3 bucket, and AWS credentials that are allowed to put objects in it.
Oobee installed. Refer to Install the portable Oobee or Run Oobee in CI/CD for the Docker image.
A scan ID, a user ID and an email address from your own system. Oobee does not create them.

Step 1: Set the upload variables

Oobee uploads results only when all four of these variables are set:

Variable	Description
`S3_BUCKET_NAME`	Name of the bucket to upload to
`OOBEE_SCAN_ID`	Unique identifier of the scan. Used in the upload path and saved as file metadata.
`OOBEE_USER_ID`	Identifier of the user. Used in the upload path and saved as file metadata.
`OOBEE_USER_EMAIL`	Email address saved as file metadata

export S3_BUCKET_NAME="<BUCKET_NAME>"
export OOBEE_SCAN_ID="<SCAN_ID>"
export OOBEE_USER_ID="<USER_ID>"
export OOBEE_USER_EMAIL="<EMAIL_ADDRESS>"
export AWS_REGION="<AWS_REGION>"

AWS_REGION is optional and defaults to ap-southeast-1. Oobee does not read AWS credentials from its own variables. Provide credentials in the way your AWS tools normally find them, for example AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY, a shared credentials file, or a role on the machine.

Set these optional variables to add more metadata to each file:

Variable	Metadata key
`OOBEE_MESSAGE_ID`	`messageid`
`OOBEE_AMPLITUDE_USER_ID`	`amplitudeuserid`
`OOBEE_DEVICE_ID`	`deviceid`
`OOBEE_ORG_ID`	`orgid`
`OOBEE_USER_ROLE`	`userrole`
`CLOUDFRONT_BASE_URL`	Not metadata. Oobee uses it as the base of the file URLs it logs.

Step 2: Run the scan

Run the scan in the same terminal, or pass the variables to the container with -e. No command-line option is needed.

npm run cli -- -c website -u <WEBSITE_URL> -k "<YOUR_NAME>:<YOUR_EMAIL>"

In Docker, pass each variable to the container:

docker run -e S3_BUCKET_NAME="<BUCKET_NAME>" -e OOBEE_SCAN_ID="<SCAN_ID>" -e OOBEE_USER_ID="<USER_ID>" -e OOBEE_USER_EMAIL="<EMAIL_ADDRESS>" -e AWS_REGION="<AWS_REGION>" <IMAGE_NAME> npm run cli -- -c website -u <WEBSITE_URL> -k "<YOUR_NAME>:<YOUR_EMAIL>"

When the scan ends, Oobee logs “Starting S3 upload…” and the upload path. If any of the four required variables is missing, Oobee logs “S3 upload not enabled (missing environment variables)” and keeps the results on disk only.

Step 3: Find the files in the bucket

Oobee uploads every file in the results folder, keeping its folder structure, under this path:

s3://<BUCKET_NAME>/users/<USER_ID>/scans/<SCAN_ID>/

Each file has the metadata keys scanid, userid and useremail, plus sitename (the title of the first page scanned) and any optional keys you set. If the scan stopped at the time limit, durationexceeded is set to true. To check the upload, list the path:

aws s3 ls s3://<BUCKET_NAME>/users/<USER_ID>/scans/<SCAN_ID>/ --recursive

Constraints and limitations

Constraint	Detail
Required variables	All four of `S3_BUCKET_NAME`, `OOBEE_SCAN_ID`, `OOBEE_USER_ID` and `OOBEE_USER_EMAIL` must be set.
AWS permissions	The credentials need permission to put objects in the bucket.
Upload failure	If an upload fails, Oobee logs the error and the scan still completes. The results stay on disk, and Oobee does not retry.
Partial uploads	Files upload in parallel. If one fails, others may already be in the bucket.
Files tracked	Oobee uploads all files. Only `.html`, `.csv`, `.pdf` and `.zip` files are listed in its upload log.
Metadata text	Oobee converts metadata values to printable ASCII, so non-English characters in a site name are removed.
npm module	Only scans started from the CLI or the interactive prompts upload to S3. The npm module functions, including `scanCustomFlow`, do not.

Run Oobee in CI/CD — run scans in a pipeline.
CLI options, environment variables and error codes — all environment variables.

Enable Safe Browsing and family DNS

How Oobee scans