supercrawler

Node.js Web Crawler

Supercrawler is a Node.js web crawler. It is designed to be highly configurable and easy to use.

When Supercrawler successfully crawls a page (which could be an image, a text document or any other file), it will fire your custom content-type handlers. Define your own custom handlers to parse pages, save data and do anything else you need.

Features

• Link Detection. Supercrawler will parse crawled HTML documents, identify links and add them to the queue.
• Robots Parsing. Supercrawler will request robots.txt and check the rules before crawling. It will also identify any sitemaps.
• Sitemaps Parsing. Supercrawler will read links from XML sitemap files, and add links to the queue.
• Concurrency Limiting. Supercrawler limits the number of requests sent out at any one time.
• Rate limiting. Supercrawler will add a delay between requests to avoid bombarding servers.
• Exponential Backoff Retry. Supercrawler will retry failed requests after 1 hour, then 2 hours, then 4 hours, etc. To use this feature, you must use the database-backed or Redis-backed crawl queue.
• Hostname Balancing. Supercrawler will fairly split requests between different hostnames. To use this feature, you must use the Redis-backed crawl queue.

How It Works

Crawling is controlled by the an instance of the Crawler object, which acts like a web client. It is responsible for coordinating with the priority queue, sending requests according to the concurrency and rate limits, checking the robots.txt rules and despatching content to the custom content handlers to be processed. Once started, it will automatically crawl pages until you ask it to stop.

The Priority Queue or UrlList keeps track of which URLs need to be crawled, and the order in which they are to be crawled. The Crawler will pass new URLs discovered by the content handlers to the priority queue. When the crawler is ready to crawl the next page, it will call the getNextUrl method. This method will work out which URL should be crawled next, based on implementation-specific rules. Any retry logic is handled by the queue.

The Content Handlers are functions which take content buffers and do some further processing with them. You will almost certainly want to create your own content handlers to analyze pages or store data, for example. The content handlers tell the Crawler about new URLs that should be crawled in the future. Supercrawler provides content handlers to parse links from HTML pages, analyze robots.txt files for Sitemap: directives and parse sitemap files for URLs.

Get Started

First, install Supercrawler.

npm install supercrawler --save

Second, create an instance of Crawler.

…

Third, add some content handlers.

…

Fourth, add a URL to the queue and start the crawl.

crawler.getUrlList()
  .insertIfNotExists(new supercrawler.Url("http://example.com/"))
  .then(function () {
    return crawler.start();
  });

That's it! Supercrawler will handle the crawling for you. You only have to define your custom behaviour in the content handlers.

Crawler

Each Crawler instance represents a web crawler. You can configure your crawler with the following options:

Example usage:

var crawler = new supercrawler.Crawler({
  interval: 1000,
  concurrentRequestsLimit: 1
});

The following methods are available:

The Crawler object fires the following events:

DbUrlList

DbUrlList is a queue backed with a database, such as MySQL, Postgres or SQLite. You can use any database engine supported by Sequelize.

If a request fails, this queue will ensure the request gets retried at some point in the future. The next request is schedule 1 hour into the future. After that, the period of delay doubles for each failure.

Options:

Example usage:

new supercrawler.DbUrlList({
  db: {
    database: "crawler",
    username: "root",
    password: "password",
    sequelizeOpts: {
      dialect: "mysql",
      host: "localhost"
    }
  }
})

The following methods are available:

RedisUrlList

RedisUrlList is a queue backed with Redis.

If a request fails, this queue will ensure the request gets retried at some point in the future. The next request is schedule 1 hour into the future. After that, the period of delay doubles for each failure.

It also balances requests between different hostnames. So, for example, if you crawl a sitemap file with 10,000 URLs, the next 10,000 URLs will not be stuck in the same host.

Options:

Example usage:

new supercrawler.RedisUrlList({
  redis: {
    host: "127.0.0.1"
  }
})

The following methods are available:

FifoUrlList

The FifoUrlList is the default URL queue powering the crawler. You can add URLs to the queue, and they will be crawled in the same order (FIFO).

Note that, with this queue, URLs are only crawled once, even if the request fails. If you need retry functionality, you must use DbUrlList.

The following methods are available:

Url

A Url represents a URL to be crawled, or a URL that has already been crawled. It is uniquely identified by an absolute-path URL, but also contains information about errors and status codes.

Example usage:

var url = new supercrawler.Url({
  url: "https://example.com"
});

You can also call it just a string URL:

var url = new supercrawler.Url("https://example.com");

The following methods are available:

handlers.htmlLinkParser

A function that returns a handler which parses a HTML page and identifies any links.

Example usage:

var hlp = supercrawler.handlers.htmlLinkParser({
  hostnames: ["example.com"]
});

var hlp = supercrawler.handlers.htmlLinkParser({
  urlFilter: function (url) {
    return url.indexOf("page1") === -1;
  }
});

handlers.robotsParser

A function that returns a handler which parses a robots.txt file. Robots.txt file are automatically crawled, and sent through the same content handler routines as any other file. This handler will look for any Sitemap: directives, and add those XML sitemaps to the crawl.

It will ignore any files that are not /robots.txt.

If you want to extract the URLs from those XML sitemaps, you will also need to add a sitemap parser.

Example usage:

var rp = supercrawler.handlers.robotsParser();
crawler.addHandler("text/plain", supercrawler.handlers.robotsParser());

handlers.sitemapsParser

A function that returns a handler which parses an XML sitemaps file. It will pick up any URLs matching sitemapindex > sitemap > loc, urlset > url > loc.

It will also handle a gzipped file, since that it part of the sitemaps specification.

Example usage:

var sp = supercrawler.handlers.sitemapsParser();
crawler.addHandler(supercrawler.handlers.sitemapsParser());

Changelog

2.0.0

• [Added] crawledurl event to contain the error message, thanks hjr3.
• [Changed] sitemapsParser to apply urlFilter on the sitemaps entries, thanks [hjr3](https://githu