Extracts URLs from OSINT Archives for Security Insights
Urx is a command-line tool designed for collecting URLs from OSINT archives, such as the Wayback Machine and Common Crawl. Built with Rust for efficiency, it leverages asynchronous processing to rapidly query multiple data sources. This tool simplifies the process of gathering URL information for a specified domain, providing a comprehensive dataset that can be used for various purposes, including security testing and analysis.
--cdx-endpoint URL, no code change needed--match-regex / --filter-regex)--meta-*): filter on first/last capture date, recorded MIME type and recorded status uniformly across every provider, after collection--scope-file): a program's own *.example.com / !admin.example.com list used verbatim, repeatable and unioned, exclusions always winning--dedup-similar)wordlist — the path segments and parameter names the target is built from, with ids, hashes and dates left out--params (the whole target's parameter inventory), --params-by-endpoint (which endpoint takes what), and --fuzz-placeholder FUZZ (one templated URL per parameter signature, ready for ffuf or dalfox)first_seen, last_seen, mime, archive_status, and digest come back with every URL a CDX archive reported, at no extra network cost--stream): URLs are written as each provider reports them, so a pipeline starts working immediately instead of waiting for the slowest archive--archive-body), so pages that no longer exist still give up the links they contained — one request per distinct body, thanks to CDX digest deduplication--expand-specs): OpenAPI 3.x, Swagger 2.0 and GraphQL introspection documents, JSON or YAML, turned into every route they describe — one request buys the whole documented surface--check-status also records Location, Content-Length and Content-Type, and --check-title adds the HTML ``--archived-discovery): every distinct version the Wayback Machine holds, so a Disallow: from 2015 still names the paths the site has since stopped mentioningurx cache subcommand to inspect and maintain the cache: stats, list, prune, drop , clear# https://crates.io/crates/urx
cargo install urx
# https://formulae.brew.sh/formula/urx
brew install urx
git clone https://github.com/hahwul/urx.git
cd urx
cargo build --release
The compiled binary will be available at target/release/urx.
urx generates its own completion script, so it always matches the flags of
the binary you actually have installed.
# zsh — any directory on your $fpath works
urx --completions zsh > ~/.zfunc/_urx
# (make sure ~/.zfunc is on the fpath, then `compinit`)
# bash
urx --completions bash > ~/.local/share/bash-completion/completions/urx
# fish
urx --completions fish > ~/.config/fish/completions/urx.fish
powershell and elvish are supported too. The flag needs no target domain.
urx --manpage > ~/.local/share/man/man1/urx.1
man urx
# Scan a single domain
urx example.com
# Scan multiple domains
urx example.com example.org
# Scan domains from a file
cat domains.txt | urx
…
--extract-links reads every URL-bearing tag, not just anchors: , , , , , ,
, , , and
targets. Relative URLs resolve against the page (honouring ``),
duplicates are collapsed, and discovered links pass through the same filters
and host validation as the rest of the run. See
docs/content/guide/cli-options.md for the
full table.
--extract-js-endpoints goes one step further and reads the JavaScript
itself: every collected URL that looks like a script is fetched and its
string literals are mined for the paths and URLs the app calls —
fetch("/api/v2/users"), axios.post("/graphql"), the static prefix of
`/api/orders/${id}`. These are the endpoints that never appear in HTML.
Output is aggressively de-noised (MIME types, module specifiers, base64,
CSS values, regex fragments and more are dropped), each body is capped at
10 MiB, the number of files fetched is bounded by --max-js-files, and the
discovered endpoints pass the same filters and host validation as everything
else. The full extraction and noise-suppression policy is in
docs/content/guide/cli-options.md.
--archive-body does the same extraction over the bodies the Wayback Machine
stored rather than over the live site, so a page that was deleted years ago
still yields the links it contained. See
Mining Archived Response Bodies.
…
--patterns / --exclude-patterns are plain substring tests: both sides are
lower-cased, and every metacharacter is a literal. --match-regex /
--filter-regex are the regex counterparts, and they differ in three ways worth
remembering:
--patterns |
--match-regex |
|
|---|---|---|
| Matching | substring | full regex syntax |
| Case | insensitive (both sides lower-cased) | sensitive — use (?i) to opt out |
| Multiple values | one comma-separated flag | repeat the flag; commas are never split |
Both regex flags are evaluated against the whole URL string as collected
(scheme, host, path, and query), so ^https:// and \.js$ both work.
Exclusion wins: a URL matching --filter-regex is dropped even if
--match-regex also matched it. A malformed expression fails the run at
startup, before any archive is queried.
A bug bounty program's scope is a list of hosts, and every platform writes it
the same way. --scope-file takes that list verbatim instead of making you
hand-translate it into anchored regex alternations — where getting the
anchoring wrong silently widens the scope rather than failing.
# scope.txt — in scope
*.example.com
api.example.org
# out of scope, even though the wildcard above covers them
!admin.example.com
!*.internal.example.com
urx example.com --subs --scope-file scope.txt
urx --domain-list targets.txt --subs --scope-file scope-a.txt --scope-file scope-b.txt
*.example.com matches the apex as well as everything under it (the
bug-bounty reading, which is what a platform's scope table means); a bare host
matches exactly that host; a lone * makes the file a pure deny-list;
exclusions always win; # starts a comment. Anything urx cannot honour — a
port, a path, a wildcard in the middle — is a startup error naming the file and
line rather than a silently wider scope. The filter applies to every provider
and to extracted links, and it combines with --strict rather than replacing
it, so a *.example.com scope line still needs --subs.
--from/--to and the --archive-* predicates are pushed into the archive's
own query, which makes them free and also limits them to CDX-backed providers —
and the two CDX dialects disagree badly enough that a positive multi-value list
(--archive-status 200,301) is unsatisfiable on pywb servers. The eight
--meta-* filters run after collection instead, over one merged set of
capture metadata per URL, so they apply to every provider uniformly.
…
URLs that carry no metadata — the non-CDX providers, --files input, cache hits
— are split by the direction of the predicate: a positive predicate cannot be
satisfied by an absent value, so the URL is dropped; an exclusion drops only
what positively matches, so it survives. --verbose reports the split, and when
missing metadata accounts for the whole result set urx says so even without
-v, because a cache hit otherwise makes an empty run look like a target with
nothing to find.
An archive will happily hand back /post/1 through /post/99999. They are one
endpoint, and --dedup-similar prints one line for them. A path segment is
treated as data — not as part of the route — when it is entirely one of:
/post/1, /page/42)/u/550e8400-e29b-41d4-a716-446655440000)/blog/2024-01-02/)Segments that merely contain digits stay put, so /api/v1/ and /api/v2/ are
still two endpoints, and a lower-case slug is prose rather than a token. Query
strings are grouped by parameter names only: ?q=cats&page=1 and
?q=dogs&page=7 collapse, while ?q=cats alone does not — dropping a
parameter changes the request.
The survivor of each group is its lexicographically smallest URL, so two runs
over the same data print the same thing. --verbose reports how many URLs were
collapsed. The option is independent of --normalize-url and
--merge-endpoint and combines with either; all three need the complete result
set, so none of them works with --stream.
--show-only-param only cuts the query string off each URL, which cannot answer
the first question a tester asks: what parameters does this target take? Three
views answer it, built on the same grouping --dedup-similar uses.
$ urx example.com --params
page
q
ref
sort
utm_source
$ urx example.com --params-by-endpoint
https://example.com/post/{id} ref,utm_source
https://example.com/search page,q,sort
$ urx example.com --fuzz-placeholder FUZZ
https://example.com/post/1?ref=FUZZ
https://example.com/post/2?utm_source=FUZZ
https://example.com/search?q=FUZZ&page=FUZZ
https://example.com/search
No open issues yet, or sync has not completed.