vessel

Vessel - high-level web crawling framework

Fast as Chrome, dead simple and yet extendable.

Vessel is a Ruby high-level web crawling framework. It can be used in a wide range of scenarios, like data mining, monitoring or historical archival. Pages are fetched by a pluggable driver: Ferrum (a real Chrome, the default) or Mechanize (plain HTTP, no browser). For automated testing we recommend Cuprite.

Install

Add this to your Gemfile:

gem "vessel"

The default :ferrum driver needs Chrome or Chromium installed and available in PATH.

The :mechanize driver is optional and ships as a sibling dependency, so add it to your Gemfile when you want to use it:

gem "mechanize"
gem "vessel"

A look around

In order to show you how Vessel works we are going to crawl together famous quotes website:

…

ruby QuotesToScrapeCom.run(delay: 2, max_threads: 1)

### domain

```ruby
class MyScraper  handler`, so different
entry points can be parsed by different methods. Without a handler `:parse` is
used:

```ruby
class MyScraper  :parse_products,
             "https://example.com/news" => :parse_news
end

If start_urls is empty the crawler is still started once with a stub response, which is handy when the first urls are computed in parse itself.

driver

class MyScraper  "text/plain",
          "Referer" => "https://example.com"
end

cookies

cookie sets one cookie, cookies takes an array of them. domain defaults to the crawler's domain:

class MyScraper  "humor"
end

You can yield a single request, an array of them, or a hash of the extracted fields, but not both in one yield.

Selectors

The response is delegated to the crawler, so inside a handler you can call directly:

• at_css, css, at_xpath, xpath - search the page
• url, data, attempt, size, body, raw - the current url, the data attached to the request, which attempt it is, the page size in bytes, the Nokogiri document and the raw html
• absolute_url, join_url, url_encode, url_decode (aliased as uri_encode, uri_decode) - url helpers

response and page are available as well, page being the driver's native object, Ferrum::Page or Mechanize::Page. response.status and response.headers give you the response status and headers, response.sync_body re-reads the html after a pause when the page needs a moment to render.

By default the selectors run against the live page, which for the Ferrum driver means Chrome does the querying. Setting Vessel.page_snapshot = true makes them run against the Nokogiri document of the page's html instead, which is much faster when you extract a lot of nodes and don't need to interact with the page.

Keep in mind that the nodes you get back come from the driver, Ferrum::Node or Nokogiri::XML::Node, and the two don't have the same api. Read the attributes with node[:href], that one works everywhere, node.attribute(:href) only works on the live Chrome page.

Fields

Instead of building a hash by hand a handler can declare fields and yield the collected fields object:

class MyScraper  :writer }` stores the field under another name

Fields can be normalized in one place by their name, put such declarations into
`config/fields` of a generated project:

```ruby
Vessel::Cargo::FieldType.add(:price) { |value| value.to_s.gsub(/[^\d.]/, "").to_f }

Every field :price is then passed through that block, typing: false skips it.

Middleware

Everything a handler yields that is not a request goes through the middleware pipeline, which runs in its own thread pool. A middleware is a class with a call(hash, fields) method, it gets the hash returned by the previous middleware plus the original fields object, and returns the hash for the next one:

class Sanitize < Vessel::Middleware
  def call(hash, fields)
    hash.transform_values { |v| v.is_a?(String) ? v.strip : v }
  end
end

class Save < Vessel::Middleware
  def call(hash, fields)
    raise Vessel::Middleware::InvalidItemError if hash[:text].to_s.empty?

    DB[:quotes].insert(hash)
    hash
  end
end

class MyScraper < Vessel::Cargo
  middleware "Sanitize", "Save"
end

Middleware is declared by name so that classes can be defined anywhere in the project, the crawler settings are available as settings. Raising Vessel::Middleware::InvalidItemError silently rejects the item, any other error is logged and counted as a rejection too.

A block passed to .run replaces the whole pipeline, which is the shortest way to collect the items in a script:

QuotesToScrapeCom.run { |item| quotes << item }

Callbacks

Override any of these in the crawler to hook into the run:

class MyScraper < Vessel::Cargo
  def before_start; end            # before anything is scheduled
  def before(stats); end           # with the initial stats
  def after_change(counter, stats) # on every stats counter change
  end
  def info(stats); end             # every 4 seconds, logs the stats by default
  def after(stats); end            # when the engine is done
  def before_stop; end             # at exit
  def on_error(request, error); end # a request failed, re-raises by default
end

The stats hash holds req_enqueued, res_dequeued, res_handled, item_pipelined, item_processed, item_sent, item_rejected and idling.

Logging

Vessel logs to $stdout at the debug level. Pass your own logger to change that:

Vessel::Logger.instance = Logger.new("log/vessel.log", level: Logger::INFO)

Projects and CLI

A single-file script is enough for a small crawler, for a bigger one Vessel can generate a project:

$ vessel new myproject
$ cd myproject
$ bundle install
$ vessel generate example.com

myproject
├── Gemfile
├── config
│   ├── boot.rb                      # loads everything below
│   ├── environments
│   │   ├── dev/dev.rb               # ApplicationCrawler for VESSEL_ENV=dev
│   │   └── prod/prod.rb             # ApplicationCrawler for VESSEL_ENV=prod
│   ├── fields                       # FieldType declarations
│   └── middleware                   # middleware classes
├── crawlers                         # one crawler per site
├── lib
│   ├── helpers
│   └── loader.rb
└── log

Crawlers inherit from ApplicationCrawler, which is defined per environment, so the settings can differ between development and production, VESSEL_ENV selects the environment and defaults to dev.

License

The gem is available as open source under the terms of the MIT License.