crawl

A simple web crawler written in shell script, designed to efficiently discover endpoints and assets within a web application.

Usage

$ crawl

$ crawl -h
|Usage:
|  crawl [-f] [href|script|form] 
|
|Options:
|  -h show help menu
|  -d number of depth to scrape.
|  -f attribute a type of link. [href|script|form]
|
|Example:
|  crawl -f script [domain].[TLD]
|  crawl -d 1 [domain].[TLD]/directory
|  crawl -f href [domain].[TLD]/directory?key=value
|
|Fetches all href, script and form links, if no flags are specified.
|Uses HTTPs as default protocol, if no protocol is specified.                                                               

$ echo google.com | crawl

Tool Chain

Get all subdomains of owasp.org and crawl the ones that are alive.

subfinder -d owasp.org | httpx | crawl

Features

• Fetches all href, script and form links.
• Highlights the Depth-1 URLs and indicates which Depth-2 URLs are included under each Depth-1 URL.
• Uses HTTPs as default protocol, if no protocol is specified.
• Depth is set to 1 by default, if depth is not specified.
• Can be linked with other tool(s)/command(s) using pipes to create a tool chain.
• Output will be color-less if it’s redirected to a file or piped to a another command.

Installation

git clone https://github.com/synacktraa/crawl.git && cd ./crawl
sudo mv ./crawl /usr/local/bin
cd .. && rm -rf "./crawl"

Dependencies

• curl
• sed
• grep
• awk
• wc
• sort
• uniq