domain-scan

A simple scanning system for the cloud

A lightweight scan pipeline for orchestrating third party tools, at scale and (optionally) using serverless infrastructure.

The point of this project is to make it easy to coordinate and parallelize third party tools with a simple scanning interface that produces consistent and database-agnostic output.

Outputs aggregate CSV for humans and machines, and detailed JSON for machines.

Can scan websites and domains for data on their HTTPS and email configuration, third party service usage, accessibility, and other things. Adding new scanners is relatively straightforward.

All scanners can be run locally using native Python multi-threading.

Some scanners can be executed inside Amazon Lambda for much higher levels of parallelization.

Most scanners work by using specialized third party tools, such as SSLyze or trustymail. Each scanner in this repo is meant to add the smallest wrapper possible around the responses returned from these tools.

There is also built-in support for using headless Chrome to efficiently measure sophisticated properties of web services. Especially powerful when combined with Amazon Lambda.

Requirements

domain-scan requires Python 3.6 or 3.7.

To install core dependencies:

pip install -r requirements.txt

You can install scanner- or gatherer-specific dependencies yourself. Or, you can "quick start" by just installing all dependencies for all scanners and/or all gatherers:

pip install -r requirements-scanners.txt
pip install -r requirements-gatherers.txt

If you plan on developing/testing domain-scan itself, install development requirements:

pip install -r requirements-dev.txt

Usage

Scan a domain. You must specify at least one "scanner" with --scan.

./scan whitehouse.gov --scan=pshtt

Scan a list of domains from a CSV. The CSV's header row will be ignored if the first cell starts with "Domain" (case-insensitive).

./scan domains.csv --scan=pshtt

Run multiple scanners on each domain:

./scan whitehouse.gov --scan=pshtt,sslyze

Append columns to each row with metadata about the scan itself, such as how long each individual scan took:

./scan example.com --scan=pshtt --meta

Scanners

• pshtt - A scanner that uses the pshtt Python package from the Department of Homeland Security's NCATS team.
• sslyze - A scanner that uses the sslyze Python package maintained by Alban Diquet.
• trustymail: The trustymail command, available from the trustymail Python package from the Department of Homeland Security's NCATS team. (Override path by setting the TRUSTYMAIL_PATH environment variable.)
• third_parties - What third party web services are in use, using headless Chrome to trap outgoing requests. (See documentation for using or writing Chrome-based scanners.)
• a11y - Accessibility issues, using pa11y.
• noop - Test scanner (no-op) used for development and debugging. Does nothing.

Parallelization

It's important to understand that scans run in parallel by default, and data is streamed to disk immediately after each scan is done.

This makes domain-scan fast, as well as memory-efficient (the entire dataset doesn't need to be read into memory), but the order of result data is unpredictable.

By default, each scanner will spin up 10 parallel threads. You can override this value with --workers. To disable this and run sequentially through each domain (1 worker), use --serial.

If row order is important to you, either disable parallelization, or use the --sort parameter to sort the resulting CSVs once the scans have completed. (Note: Using --sort will cause the entire dataset to be read into memory.)

Lambda

The domain-scan tool can execute certain compatible scanners in Amazon Lambda, instead of locally.

This can allow the use of hundreds of parallel workers, and can speed up large scans by orders of magnitude. (Assuming that the domains you're scanning are disparate enough to avoid DDoS-ing any particular service!)

See docs/lambda.md for instructions on configuring scanners for use with Amazon Lambda.

Once configured, scans can be run in Lambda using the --lambda flag, like so:

./scan example.com --scan=pshtt,sslyze --lambda

Headless Chrome

This tool has some built-in support for instrumenting headless Chrome, both locally and inside of Amazon Lambda.

Install a recent version of Node (using a user-space version manager such as nvm or nodeenv is recommended).

Then install dependencies:

npm install

Chrome-based scanners use Puppeteer, a Node-based wrapper for headless Chrome that is maintained by the Chrome team. This means that Chrome-based scanners make use of Node, even while domain-scan itself is instrumented in Python. This makes initial setup a little more complicated.

• During local scans, Python will shell out to Node from ./scanners/headless/local_bridge.py by executing ./scanners/headless/local_bridge.js, which expects the /usr/bin/env node to be usable as its executor. The data which is sent into the Node scanner, including original CLI options and environment data, is passed as a serialized JSON string as a CLI parameter, and the Node scanner returns data back to Python by emitting JSON over STDOUT.

• During Lambda scans, local execution remains exclusively in Python, and Node is never used locally. However, the Lambda function itself is expected to be in the node6.10 runtime, and uses a special Node-based Lambda handler in lambda/headless/handler.js for this purpose. There is a separate lambda/headless/deploy script for the building and deployment of Node/Chrome-based Lambda functions.

It is recommended to use Lambda in production for Chrome-based scanners -- not only for the increased speed, but because they use a simpler and cleaner method of cross-language communication (the HTTP-based function call to Amazon Lambda itself).

Support for running headless Chrome locally is intended mostly for testing and debugging with fewer moving parts (and without risk of AWS costs). Lambda support is the expected method for production scanning use cases.

See below for how to structure a Chrome-based scanner.

See docs/lambda.md for how to build and deploy Lambda-based scanners.

Options

General options:

• --scan - Required. Comma-separated names of one or more scanners.
• --sort - Sort result CSVs by domain name, alphabetically. (Note: this causes the entire dataset to be read into memory.)
• --serial - Disable parallelization, force each task to be done simultaneously. Helpful for testing and debugging.
• --debug - Print out more stuff. Useful with --serial.
• --workers - Limit parallel threads per-scanner to a number.
• --output - Where to output the cache/ and results/ directories. Defaults to ./.
• --cache - Use previously cached scan data to avoid scans hitting the network where possible.
• --suffix - Add a suffix to all input domains. For example, a --suffix of virginia.gov will add .virginia.gov to the end of all input domains.
• --lambda - Run certain scanners inside Amazon Lambda instead of locally. (See the Lambda instructions for how to use this.)
• --lambda-profile - When running Lambda-related commands, use a specified AWS named profile. Credentials/config for this named profile should already be configured separately in the execution environment.
• --meta - Append some additional columns to each row with information about the scan itself. This includes start/end times and durations, as well as any encountered errors. When also using --lambda, additional Lambda-specific information will be appended.

Output

All output files are placed into cache/ and results/ directories, whose location defaults to the current directory (./). Override the output home with --output.

• Cached full scan data about each domain is saved in the cache/ directory, named after each scan and each domain, in JSON.

Example: cache/pshtt/whitehouse.gov.json

• Formal output data in CSV form about all domains are saved in the results/ directory in CSV form, named after each scan.

Example: results/pshtt.csv

You can override the output directory by specifying --output.

It's possible for scans to save multiple CSV rows per-domain. For example, the a11y scan will have a row with details for each detected accessibility error.

• Scan metadata with the start time, end time, and scan command will be placed in the results/ directory as meta.json.

Example: results/meta.json

Using with Docker

If you're using Docker Compose, run:

docker-compose up

(You may need to use sudo.)

To scan, prefix commands with docker-compose run:

docker-compose run scan  --scan=

Gathering hostnames

This tool also includes a facility for gathering domain names that end in one or more given suffixes (e.g. .gov or yahoo.com or .gov.uk) from various sources.

By default, only fetches third-level and higher domains (excluding second-level domains).

Usage:

./gather [source] [options]

Or gather hostnames from multiple sources separated by commas:

./gather [source1,source2,...,sourceN] [options]

Right now there's one specific source (Censys.io), and then a general way of sourcing URLs or files by whatever name is convenient.

Censys.io - The censys gatherer uses data from Censys.io, which has hostnames gathered from observed certificates, through the Google BigQuery API. Censys provides certificates observed from a nightly zmap scan of the IPv4 space, as well as certificates published to public Certificate Transparency logs.

Remote or local CSV - By using any other name besides censys, this will define a gatherer based on an HTTP/HTTPS URL or local path to a CSV. Its only option is a flag named after itself. For example, using a gatherer name of dap will mean that domain-scan expects --dap to point to the URL or local file.

Hostnames found from multiple sources are deduped, and filtered by suffix or base domain according to the options given.

The resulting gathered.csv will have the following columns:

• the hostname
• the hostname's base domain
• one column for each checked source, with a value of True/False based on the hostname's presence in each source

See specific usage examples below.

General options:

• --suffix: Required. One or more suffix to filter on, separated by commas as necessary. (e.g. .gov or .yahoo.com or .gov,.gov.uk)
• --parents: A path or URL to a CSV whose first column is second-level domains. Any subdomain not contained within these second-level domains will be excluded.
• --include-parents: Include second-level domains. (Defaults to false.)
• --ignore-www: Ignore the www. prefixes of hostnames. If www.staging.example.com is found, it will be treated as staging.example.com.
• --debug: display extra output

censys: Data from Censys.io via Google BigQuery

Gathers hostnames from