百科.dev
全部条目在线工具AI 编程趋势榜开源项目技术资讯提交条目
登录
< 返回工具列表
H

harvestman

> 编程语言
开源

快速而脏乱的 Web 爬虫。

273 stars0 点赞2 次浏览
访问官网GitHub

工具介绍

快速而脏乱的 Web 爬虫。

Harvestman

Harvestman is a very simple, lightweight web crawler for Quick'n'Dirty™ web scraping.

It's quite useful for scraping search result pages:

ruby
require 'harvestman'

Harvestman.crawl 'http://www.foo.com/bars?page=*', (1..5) do
  price = css 'div.item-price a'
  ...
end

[!] Warning: this gem is in alpha stage (no tests), don't use it for anything serious.

Installation

Via command line:

$ gem install harvestman

Basic usage

Harvestman is fairly simple to use: you specify the URL to crawl and pass in a block. Inside the block you can call the css (or xpath) method to search the HTML document and get the inner text inside each node. See Nokogiri for more information.

Perhaps this is best understood with an example:
ruby
Harvestman.crawl "http://www.24pullrequests.com" do
  headline = xpath "//h3"
  catchy_phrase = css "div.visible-phone h3"

  puts "Headline: #{headline}"
  puts "Catchy phrase: #{catchy_phrase}"
end

One node to rule them all

Harvestman assumes there's only one node at the path you passed to the css. If there is more than one node at that path, you can pass in an additional block.

Another example:
ruby
Harvestman.crawl 'http://en.wikipedia.org/wiki/Main_Page' do
  # Print today's featured article
  tfa = css "div#mp-tfa"

  puts "Today's featured article: #{tfa}"

  # Print all the sister projects
  sister_projects = []

  css "div#mp-sister b" do
    sister_projects << css("a")
  end

  puts "Sister projects:"
  sister_projects.each { |sp| puts "- #{sp}" }
end

Note that inside the block we use css("a") and not css("div#mp-sister b a"). Calls to css or xpath here assume div#mp-sister b is the parent node.

Pages / Search results

If you want to crawl a group of similar pages (eg: search results, as shown above), you can insert a * somewhere in the URL string and it will be replaced by each element in the second argument.

Final example:
ruby
require 'harvestman'

Harvestman.crawl 'http://www.etsy.com/browse/vintage-category/electronics/*', (1..3) do
  css "div.listing-hover" do
    title = css "div.title a"
    price = css "span.listing-price"

    puts "* #{title} (#{price})"
  end
end

The above code is going to crawl Etsy's electronics category pages (from 1 to 3) and output every item's title and price. Here we're using a range (1..3) but you could've passed an array with search queries:

"http://www.site.com?query=*", ["dogs", "cats", "birds"]

Performance

When using the * feature described above, each page is run inside a separate thread. You can disable multithreading by passing an additional argument :plain to the crawl method, like this:

ruby
require 'harvestman'

Harvestman.crawl 'http://www.store.com/products?page=*', (1..99), :plain do
  ...
end

Needless to say, this will greatly decrease performance.

License

See LICENSE.txt

Issues· 2 开放

查看全部 Issues在 GitHub 打开

暂无开放 Issues,或尚未同步最近议题。

> 标签

Ruby

暂无评论,来聊聊你的看法吧

> 工具信息

发布日期2026年8月1日
最后更新2026年9月18日
分类编程语言
定价开源

> 相关工具

T
TypeScript
JavaScript 的超集,为前端与全栈提供静态类型
P
Python
通用编程语言,广泛用于 Web、数据与 AI
G
Go
Google 推出的简洁高效系统语言