gammo

一个纯粹的 Ruby HTML5 兼容解析器,具有 CSS 选择器和 XPath 1.0 遍历功能

Gammo - A pure-Ruby HTML5 parser

Gammo provides a pure Ruby HTML5-compliant parser and CSS selector / XPath support for traversing the DOM tree built by Gammo. The implementation of the HTML5 parsing algorithm in Gammo conforms the WHATWG specification. Given an HTML string, Gammo parses it and builds DOM tree based on the tokenization and tree-construction algorithm defined in WHATWG parsing algorithm, these implementations are provided without any external dependencies.

Gammo, its naming is inspired by Gumbo. But Gammo is a fried tofu fritter made with vegetables.

require 'gammo'
require 'open-uri'

parser = URI.open('https://google.com') { |f| Gammo.new(f.read) }
document = parser.parse #=> #

puts document.css('title').first.inner_text #=> 'Google'

Overview
- Features
Tokenizaton
- Token types
Parsing
- Notes
Node
DOM Tree Traversal
- XPath 1.0 (experimental)
- CSS Selector (experimental)
Performance
References
License
Release History

Overview

Features

Tokenization: Gammo has a tokenizer for implementing the tokenization algorithm.
Parsing: Gammo provides a parser which implements the parsing algorithm by the above tokenization and the tree-construction algorithm.
Node: Gammo provides the nodes which implement WHATWG DOM specification partially.
DOM Tree Traversal: Gammo provides a way of DOM tree traversal (CSS selector / XPath).
Performance: Gammo does not prioritize performance, and there are a few potential performance notes.

Tokenizaton

Gammo::Tokenizer implements the tokenization algorithm in WHATWG. You can get tokens in order by calling Gammo::Tokenizer#next_token.

Here is a simple example for performing only the tokenizer.

def dump_for(token)
  puts "data: #{token.data}, class: #{token.class}"
end

tokenizer = Gammo::Tokenizer.new('')
dump_for tokenizer.next_token #=> data: html, class: Gammo::Tokenizer::DoctypeToken
dump_for tokenizer.next_token #=> data: input, class: Gammo::Tokenizer::StartTagToken
dump_for tokenizer.next_token #=> data: frameset, class: Gammo::Tokenizer::StartTagToken
dump_for tokenizer.next_token #=> data: end of string, class: Gammo::Tokenizer::ErrorToken

The parser described below depends on this tokenizer, it applies the WHATWG parsing algorithm to the tokens extracted by this tokenization in order.

Token types

The tokens generated by the tokenizer will be categorized into one of the following types:

  Token type
  Description






  Gammo::Tokenizer::ErrorToken
  Represents an error token, it usually means end-of-string.




  Gammo::Tokenizer::TextToken
  Represents a text token like "foo" which is inner text of elements.




  Gammo::Tokenizer::StartTagToken
  Represents a start tag token like &lt;a&gt;.




  Gammo::Tokenizer::EndTagToken
  Represents an end tag token like &lt;/a&gt;.




  Gammo::Tokenizer::SelfClosingTagToken
  Represents a self closing tag token like &lt;img /&gt;




  Gammo::Tokenizer::CommentToken
  Represents a comment token like &lt;!-- comment --&gt;.




  Gammo::Tokenizer::DoctypeToken
  Represents a doctype token like &lt;!doctype html&gt;.

Parsing

Gammo::Parser implements processing in the tree-construction stage based on the tokenization described above.

A successfully parsed parser has the document accessor as the root document (this is the same as the return value of the Gammo::Parser#parse). From the document accessor, you can traverse the DOM tree constructed by the parser.

require 'gammo'
require 'pp'

document = Gammo.new('').parse

def dump_for(node, strm)
  strm ').parse
node_set = document.xpath('//input[@type="button"]') #=> ""

node_set.length #=> 1
node_set.first #=> ""

Since this is implemented by full scratch, Gammo is providing this support as a very experimental feature. Please file an issue if you find bugs.

Example

Before proceeding at the details of XPath support, let's have a look at a few simple examples. Given a sample HTML text and its DOM tree:

document = Gammo.new(

  
namusyaka.com

  
Here is a sample web site.

  

    
hello

    
world

  

  

    
Google google.com

    
GitHub github.com/namusyaka

  

EOS

The following XPath expression gets all li elements and prints those text contents:

document.xpath('//li').each do |elm|
  puts elm.inner_text
end

The following XPath expression gets all li elements under the ul element having the id=links attribute:

document.xpath('//ul[@id="links"]/li').each do |elm|
  puts elm.inner_text
end

The following XPath expression gets each text node for each li element under the ul element having the id=links attribute:

document.xpath('//ul[@id="links"]/li/text()').each do |elm|
  puts elm.data
end

Axis Specifiers

In the combination with Gammo, the axis specifier indicates navigation direction within the DOM tree built by Gammo. Here is list of axes. As you can see, Gammo fully supports the all of axes.

  Full Syntax
  Abbreviated Syntax
  Supported
  Notes






  ancestor
  
  yes
  




  ancestor-or-self
  
  yes
  




  attribute
  @
  yes
  @abc is the alias for attribute::abc




  child
  
  yes
  abc is the short for child::abc




  descendant
  
  yes
  




  descendant-or-self
  //
  yes
  // is the alias for /descendant-or-self::node()/




  following
  
  yes
  




  following-sibling
  
  yes
  




  namespace
  
  yes
  




  parent
  ..
  yes
  .. is the alias for parent::node()




  preceding
  
  yes
  




  preceding-sibling
  
  yes
  




  self
  .
  yes
  . is the alias for self::node()

Node Test

Node tests consist of specific node names or more general expressions. Although particular syntax like : should work for specifying namespace prefix in XPath, Gammo does not support it yet as it's not a core feature in HTML5.

  Full Syntax
  Supported
  Notes






  text()
  yes
  Finds a node of type text, e.g. hello in &lt;p&gt;hello &lt;a href="https://hello"&gt;world&lt;/a&gt;&lt;/p&gt;




  comment()
  yes
  Finds a node of type comment, e.g. &lt;!-- comment --&gt;




  node()
  yes
  Finds any node at all.

Also note that the processing-instruction is not supported. There is no plan to support it.

Operators

The /, // and [] are used in the path expression.
The union operator | forms the union of two node sets.
The boolean operators: and, or
The arithmetic operators: +, -, *, div and mod
Comparison operators: =, !=, ``, =

Functions

XPath 1.0 defines four data types (nodeset, string, number, boolean) and there are various functions based on the types. Gammo supports those functions partially, please check it to be supported before using functions.

Node set functions

  Function Name
  Supported
  Specification






  last()
  yes
  https://www.w3.org/TR/1999/REC-xpath-19991116/#function-last




  position()
  yes
  https://www.w3.org/TR/1999/REC-xpath-19991116/#function-position




  count(node-set)
  yes
  https://www.w3.org/TR/1999/REC-xpath-19991116/#function-count

String Functions

  Function Name
  Supported
  Specification






  string(object?)
  yes
  https://www.w3.org/TR/1999/REC-xpath-19991116/#function-string




  concat(string, string, string*)
  yes
  https://www.w3.org/TR/1999/REC-xpath-19991116/#function-concat




  starts-with(string, string)
  yes
  https://www.w3.org/TR/1999/REC-xpath-19991116/#function-starts-with




  contains(string, string)
  yes
  https://www.w3.org/TR/1999/REC-xpath-19991116/#function-contains




  substring-before(string, string)
  yes
  https://www.w3.org/TR/1999/REC-xpath-19991116/#function-substring-before




  substring-after(string, string)
  yes
  https://www.w3.org/TR/1999/REC-xpath-19991116/#function-substring-after




  substring(string, number, number?)
  no
  https://www.w3.org/TR/1999/REC-xpath-19991116/#function-substring




  string-length(string?)
  no
  https://www.w3.org/TR/1999/REC-xpath-19991116/#function-string-length




  normalize-space(string?)
  no
  https://www.w3.org/TR/1999/REC-xpath-19991116/#function-string-normalize-space




  translate(string, string, string)
  no
  https://www.w3.org/TR/1999/REC-xpath-19991116/#function-string-translate

Boolean Functions

  Function Name
  Supported
  Specification






  boolean(object)
  yes
  https://www.w3.org/TR/1999/REC-xpath-19991116/#function-boolean




  not(object)
  yes
  https://www.w3.org/TR/1999/REC-xpath-19991116/#function-not




  true()
  yes
  https://www.w3.org/TR/1999/REC-xpath-19991116/#function-true




  false()
  yes
  https://www.w3.org/TR/1999/REC-xpath-19991116/#function-false




  lang()
  no
  https://www.w3.org/TR/1999/REC-xpath-19991116/#function-lang

Number Functions

  Function Name
  Supported
  Specification






  number(object?)
  no
  https://www.w3.org/TR/1999/REC-xpath-19991116/#function-number




  sum(node-set)
  no
  https://www.w3.org/TR/1999/REC-xpath-19991116/#function-sum




  floor(number)
  no
  https://www.w3.org/TR/1999/REC-xpath-19991116/#function-floor




  ceiling(number)
  yes
  https://www.w3.org/TR/1999/REC-xpath-19991116/#function-ceiling




  round(number

gammo

工具介绍

Gammo - A pure-Ruby HTML5 parser

Overview

Features

Tokenizaton

Token types

Parsing

Example

Axis Specifiers

Node Test

Operators

Functions

Node set functions

String Functions

Boolean Functions

Number Functions

Issues· 0 开放

> 标签

> 工具信息

> 相关工具

gammo

工具介绍

Gammo - A pure-Ruby HTML5 parser

Overview

Features

Tokenizaton

Token types

Parsing

Example

Axis Specifiers

Node Test

Operators

Functions

Node set functions

String Functions

Boolean Functions

Number Functions

Issues· 0 开放

> 标签

> 讨论区

> 工具信息

> 相关工具