百科.dev
全部条目在线工具AI 编程趋势榜开源项目技术资讯提交条目
登录
< 返回工具列表
D

data_miner

> 编程语言
开源

下载、从 ZIP/TAR/GZ/BZ2 存档中解压、解析、校正、转换单位,并将 Google 电子表格、XLS、ODS、XML、CSV、HTML 等导入到您的 ActiveRec 中

305 stars0 点赞0 次浏览
访问官网GitHub

工具介绍

下载、从 ZIP/TAR/GZ/BZ2 存档中解压、解析、校正、转换单位,并将 Google 电子表格、XLS、ODS、XML、CSV、HTML 等导入到您的 ActiveRec 中

data_miner

Download, pull out of a ZIP/TAR/GZ/BZ2 archive, parse, correct, and import XLS, ODS, XML, CSV, HTML, etc. into your ActiveRecord models.

Tested in MRI 1.8.7+, MRI 1.9.2+, and JRuby 1.6.7+. Thread safe.

Real-world usage

We use data_miner for data science at Brighter Planet and in production at

  • Brighter Planet's reference data web service
  • Brighter Planet's impact estimate web service

The killer combination for us is:

  1. active_record_inline_schema - define table structure
  2. remote_table - download data and parse it
  3. errata - apply corrections in a transparent way
  4. data_miner (this library!) - import data idempotently

Documentation

Check out the extensive documentation.

Quick start

You define data_miner blocks in your ActiveRecord models. For example, in app/models/country.rb:

class Country  :integer # numeric like 826; aka UN M49 code
  col :iso_3166_alpha_3_code                    # 3-letter like GBR
  col :name

  data_miner do
    # auto_upgrade! is provided by active_record_inline_schema
    process :auto_upgrade!

    import("OpenGeoCode.org's Country Codes to Country Names list",
           :url => 'http://opengeocode.org/download/countrynames.txt',
           :format => :delimited,
           :delimiter => '; ',
           :headers => false,
           :skip => 22) do
      key   :iso_3166_code, :field_number => 0
      store :iso_3166_alpha_3_code, :field_number => 1
      store :iso_3166_numeric_code, :field_number => 2
      store :name, :field_number => 5
    end
  end
end

Now you can run:

>> Country.run_data_miner!
=> nil

More advanced usage

The earth library has dozens of real-life examples showing how to download, pull out of a ZIP/TAR/BZ2 archive, parse, correct, and import CSVs, fixed-width files, ODS, XLS, XLSX, even HTML and XML:

Model
Highlights
Reference




Aircraft
parsing Microsoft Frontpage HTML (!)
data_miner.rb




Airports
forcing column names and use of :select block (Proc)
data_miner.rb




Automobile model variants
super advanced usage of "custom parser" and errata
data_miner.rb




Country
parsing CSV and a few other tricks
data_miner.rb




EGRID regions
parsing XLS
data_miner.rb




Flight segment (stage)
super advanced usage of POSTing form data
data_miner.rb




Zip codes
downloading a ZIP file and pulling an XLSX out of it
data_miner.rb

And many more - look for the data_miner.rb file that corresponds to each model. Note that you would normally put the data_miner declaration right inside the ActiveRecord model file... it's kept separate in earth so that loading it is optional.

Authors

  • Seamus Abshere
  • Andy Rossmeissl
  • Derek Kastner
  • Ian Hough
  • Tower He

Wishlist

  • Make the tests real unit tests
  • sql steps shouldn't shell out if binaries are missing

Copyright

Copyright (c) 2013 Seamus Abshere

Issues· 8 开放

查看全部 Issues在 GitHub 打开

暂无开放 Issues,或尚未同步最近议题。

> 标签

Ruby

暂无评论,来聊聊你的看法吧

> 工具信息

发布日期2026年8月1日
最后更新2026年9月18日
分类编程语言
定价开源

> 相关工具

T
TypeScript
JavaScript 的超集,为前端与全栈提供静态类型
P
Python
通用编程语言,广泛用于 Web、数据与 AI
G
Go
Google 推出的简洁高效系统语言