simple multi-level scraper json input/output for Cheerio
simple multi-level scraper json input/output for Cheerio
jsonframe
simple multi-level scraper json input/outputJSON Syntax: input json, output the same structured json including with scraped data
Simple patterns: simple inline selectors, extractors, filters and parser.
Reliable & fast: used in production within crawlers
let cheerio = require('cheerio')
let $ = cheerio.load(`
I love jsonframe!
Email: [email protected]
`)
let jsonframe = require('jsonframe-cheerio')
jsonframe($) // initializing the plugin
let frame = {
"title": "h1", // this is an inline selector
"email": "span[itemprop=email]
{
"title": "I love jsonframe!",
"email": "gabin@datascraper.pro"
}
/*Install the plugin to your Node.js app through NPM
npm i jsonframe-cheerio --saveStart by loading Cheerio.
let cheerio = require('cheerio')
let $ = cheerio.load("HTML DOM to load") // See Cheerio APIThen load the jsonframe plugin.
let jsonframe = require('jsonframe-cheerio') // require from npm package
jsonframe($) // apply the plugin to the current Cheerio instanceOnce the plugin is loaded, you've first got to set the frame of your data.
Let's take the following HTML example:
Pricing
A Link
We are the 04/02/2017
Hacker
Free
Pro
$39
Phone FR: +332 38 30 37 90
Email: [email protected]
selector is defined in Cheerio's documentation
frame is a JSON or Javascript Object
{options} are detailed later in its own section
let frame = {
"title": "h2" // CSS selector
}We then pass the frame to the function:
let result = $('body').scrape(frame, { string: true })
console.log( result )
//=> {"title": "Pricing"}Most common selector, inline line by specifying nothing more than the data name property and the selector as its value.
...
let frame = { "title": "h2" }
let result = $('body').scrape(frame, { string: true })
console.log( result )
/* output =>
{ "title": "Pricing" }
*/
...You can now declare everything in line. You should just be careful to always use them in the following order when combining them : @ (attribute), | (extractor), || (parse).
See examples for each of them above.
_a: "attributeName" allows you to retrieve any attribute data@ inside the selector _s allows you to do it inline
...
let frame = {
"proPrice": ".planName:contains('Pro') + span@price"
}
let result = $('body').scrape(frame, { string: true })
console.log( result )
/* output =>
{ "proPrice": "39.00" }
*/
...` { "email": "[email protected]", "frphone": "33238303790" } */ ...
#### Filter
`|` inside the selector `_s` allows you to do it inline
It currently supports `trim` (remove spaces at beginning and end), `lowercase or lcase`, `uppercase or ucase`, `capitalize or cap`, `words or w`, `noescapchar or nec`, `compact or cmp` and `number or nb`.
```js
...
let frame = {
"email1": "[itemprop=email]
{
"email1": "[email protected]",
"email2": "EXAMPLE GOOGLE NET"
}
*/
...|| inside the selector _s allows you to use regexes in line
_p: /regex/ allows you to extract data based on regular expressions
...
let frame = {
"data": ".date || \\d{1,2}/\\d{1,2}/\\d{2,4}"
}
// or use the longer version for proper regex entry
let frame = {
"data": {
_s: ".date",
_p: /\d{1,2}\/\d{1,2}\/\d{2,4}/ // n[n]/n[n]/nn[nn] format here
}
}
let result = $('body').scrape(frame, { string: true })
console.log( result )
/* output =>
{
"date": "04/02/2017"
}
*/
..._d: [{ }] allows you to get an array / list of data_d: ["selector"] will retrieves a list based on the selector inbetween quotes._d: ["firstSelector", "secondSelector"] works too and merge the results into one array
You could even shorten it more by listing right from the selector as follows:
"selectorName": [".selector"] which returns an array of strings
...
let frame = {
"pricing": {
_s: "#pricing .item",
_d: [{
"name": ".planName",
"price": ".planPrice"
}]
}
}
let result = $('body').scrape(frame, { string: true })
console.log( result )
/* output =>
{
"pricing": [
{
"name": "Hacker",
"price": "Free"
},
{
"name": "Pro",
"price": "$39"
}
]
}
*/
// Or a shorter way which works for simple string arrays
let frame = {
"pricingNames": ["#pricing .item .planName"]
}
let result = $('body').scrape(frame, { string: true })
console.log( result )
/* output =>
{
"pricingNames": ["Hacker", "Pro"]
}
*/
..."_g": { _s: "", _d: {} } allows you to group some data selectors by a parent selector without naming the parent. You can also extends the group property to add some meaning or simply have several groups at the same level.
Group property name must be _g or _group followed by _ and whatever string you want.
ex: _g_head : {} or _g_body : {}
...
let frame = {
_g: {
_s: "#pricing .item",
_d: {
"name": ".planName",
"price": ".planPrice"
}
},
_g_second: {
_s: "#pricing .item",
_d: {
"secondName": ".planName",
"secondPrice": ".planPrice"
}
}
}
let result = $('body').scrape(frame, { string: true })
console.log( result )
/* output =>
{
"name": "Hacker",
"price": "Free",
"secondName": "Hacker",
"secondPrice": "Free"
}
*/
..."parent": { _s: "parentSelector", _d: {} } allows you to segment your data by setting a parent section from which the child data will be scraped.
You can also use "parent": { } when you only want to nest data into objects without setting a parent selector.
...
let frame = {
"pricing": {
_s: "#pricing .item",
_d: {
"name": ".planName",
"price": ".planPrice"
}
}
}
let result = $('body').scrape(frame, { string: true })
console.log( result )
/* output =>
{
"pricing":{
"name": "Hacker",
"price": "Free"
}
}
*/
...Note here that we get the first returned result (#pricing .item).
See how you can properly structure your data, ready for the output!
...
let frame = {
"pricing": {
_s: "#pricing .item",
_d: [{
"name": ".planName",
"price": ".planPrice @ price",
"image": {
"url": "img @ src",
"link": "a @ href"
}
}]
}
}
let result = $('body').scrape(frame, { string: true })
console.log( result )
/* output =>
{
"pricing":[
{
"name": "Hacker",
"price": "0",
"image": {
"url": "./img/hacker.png",
"link": "/hacker"
}
},
{
"name": "Pro",
"price": "39.00",
"image": {
"url": "./img/pro.png",
"link": "/pro"
}
}
]
}
*/
...Note here that we get the first returned result (#pricing .item).
...
let frame = {
"proPrice": {
_s: ".planName:contains('Pro') + span",
_a: "price"
}
}
let result = $('body')
.scrape(frame, {
timestats: true, // default: false
string: true // default: false
})
console.log(result)
/* output =>
{
"proPrice": {
"value":"39.00",
"_timestats": "1" // ms
}
}
*/
...One shot tests
npm run testWatching test on updates
npm run test-watch⚠ Careful if you've been using jsonframe from the version 1.x.x, some things changed to make it more flexible, faster to use (inline parameters) and more meaningful in the syntax.
2.0.52 (28/02/2017)
2.0.51 (27/02/2017)
2.0.50 (27/02/2017)
issue so we get the discussion startedgit checkout -b my-new-featuregit commit -am 'Add some feature'git push origin my-new-featureGabin Desserprit - datascraper.pro
Released under MIT License
No open issues yet, or sync has not completed.