gospider

# GoSpider **GoSpider** - Fast web spider written in Go ## Painless integrate Gospider into your recon workflow?

this project was part of Osmedeus Engine. Check out how it was integrated at @OsmedeusEngine

## Installation ### GO install ``` GO111MODULE=on go install github.com/jaeles-project/gospider@latest ``` ### Docker ```bash # Clone the repo git clone https://github.com/jaeles-project/gospider.git # Build the contianer docker build -t gospider:latest gospider # Run the container docker run -t gospider -h ``` ## Features * Fast web crawling * Brute force and parse sitemap.xml * Parse robots.txt * Generate and verify link from JavaScript files * Link Finder * Find AWS-S3 from response source * Find subdomains from response source * Get URLs from Wayback Machine, Common Crawl, Virus Total, Alien Vault * Format output easy to Grep * Support Burp input * Crawl multiple sites in parallel * Random mobile/web User-Agent ## Showcases ## Usage ``` … ``` ## Example commands ### Quite output ``` gospider -q -s "https://google.com/" ``` #### Run with single site ``` gospider -s "https://google.com/" -o output -c 10 -d 1 ``` #### Run with site list ``` gospider -S sites.txt -o output -c 10 -d 1 ``` #### Run with 20 sites at the same time with 10 bot each site ``` gospider -S sites.txt -o output -c 10 -d 1 -t 20 ``` #### Also get URLs from 3rd party (Archive.org, CommonCrawl.org, VirusTotal.com, AlienVault.com) ``` gospider -s "https://google.com/" -o output -c 10 -d 1 --other-source ``` #### Also get URLs from 3rd party (Archive.org, CommonCrawl.org, VirusTotal.com, AlienVault.com) and include subdomains ``` gospider -s "https://google.com/" -o output -c 10 -d 1 --other-source --include-subs ``` #### Use custom header/cookies ``` gospider -s "https://google.com/" -o output -c 10 -d 1 --other-source -H "Accept: */*" -H "Test: test" --cookie "testA=a; testB=b" gospider -s "https://google.com/" -o output -c 10 -d 1 --other-source --burp burp_req.txt ``` #### Blacklist url/file extension. **P/s**: gospider blacklisted `.(jpg|jpeg|gif|css|tif|tiff|png|ttf|woff|woff2|ico)` as default ``` gospider -s "https://google.com/" -o output -c 10 -d 1 --blacklist ".(woff|pdf)" ``` #### Show and Blacklist file length. ``` gospider -s "https://google.com/" -o output -c 10 -d 1 --length --filter-length "6871,24432" ``` ## License `Gospider` is made with ♥ by [@j3ssiejjj](https://twitter.com/j3ssiejjj) & [@thebl4ckturtle](https://twitter.com/thebl4ckturtle) and it is released under the MIT license. ## Donation