Web Application Analysisweb-spidercrawlingsitemaprobots-txtjavascriptsubdomainsaws-s3wayback-machine

Gospider

Gospider is a fast web spider written in Go for crawling websites. It supports features like parsing sitemaps and robots.txt, generating links from JavaScript, and extracting URLs from various sources.

Description

Gospider is designed for rapid web crawling and reconnaissance tasks in cybersecurity. It performs brute force parsing of sitemap.xml files, extracts directives from robots.txt, and generates verifiable links from JavaScript files. The tool also includes a Link Finder capability and can identify AWS S3 buckets and subdomains directly from response sources.

Use cases include reconnaissance for web application testing, where it fetches URLs from archives like Wayback Machine, Common Crawl, Virus Total, and Alien Vault. Its output is formatted for easy grepping, supports Burp Suite input, and allows parallel crawling of multiple sites with randomized User-Agents for evasion.

As a Kali Linux tool, Gospider aids in information gathering by providing comprehensive site mapping and link discovery, making it valuable for penetration testers mapping attack surfaces efficiently.

How It Works

Gospider operates as a multi-threaded web crawler using Go's concurrency model for fast site traversal. It sends HTTP requests with configurable User-Agents (random web/mobile or custom), proxies, and follows links while parsing structured data like sitemap.xml and robots.txt. JavaScript files are analyzed to extract and verify URLs, response bodies are scanned for AWS S3 patterns and subdomains, and external sources like Wayback Machine are queried for historical URLs. Output is structured in a grep-friendly format to a specified folder.

Installation

bash
sudo apt install gospider

Flags

-s, --siteSite to crawl
-S, --sitesSite list to crawl
-p, --proxyProxy (Ex: http://127.0.0.1:8080)
-o, --outputOutput folder
-u, --user-agentUser Agent to use (web: random web user-agent, mobi: random mobile user-agent, or custom; default "web")

Examples

Display help and usage information for the tool
gospider -h
Crawl a single specified site
gospider -s https://example.com
Crawl multiple sites listed in a file
gospider -S sites.txt
Crawl a site through a specified proxy like Burp Suite
gospider -s https://example.com -p http://127.0.0.1:8080
Crawl a site and save output to a folder
gospider -s https://example.com -o output_dir
Crawl a site using random mobile User-Agent
gospider -s https://example.com -u mobi
Crawl a site with a custom User-Agent string
gospider -s https://example.com -u "Mozilla/5.0 (Custom UA)"
Updated 2026-04-16kali.org ↗