Web 爬虫和 HTML 重新处理。简单易用。
Web scraping and HTML-reprocessing. The easy way.
ineed allows you collect useful data from web pages using simple and nice API. Let's collect images, hyperlinks, scripts and stylesheets from http://google.com:
var ineed = require('ineed');
ineed.collect.images.hyperlinks.scripts.stylesheets.from('http://google.com',
function (err, response, result) {
console.log(result);
});
Also, it can be used to build HTML-reprocessing pipelines (like jch/html-pipeline but for Node) with elegance. E.g. we have the following html:
This is an H1
=============
* Red
* Green
* Blue
Let's render it's `
This is an H1
Red
Green
Blue
…$ npm install ineed
## Usage
The general form:
#### ```ineed.[....].```
### from* methods
##### .fromHtml(html)
Accepts `html` string as an argument and synchronously returns `result` of the action.
*Example:*
```js
var result = ineed.collect.texts.fromHtml('
');Asynchronously loads specified page and invokes callback(err, response, result). It passes response object to callback in case if you need response headers or status codes. The first argument can be either a url or an options object.
The set of options is the same as in @mikeal's request, so you can use POST method, set request headers or do any other advanced setup for the page request.
Examples:
ineed.collect.jsCode.from('https://github.com', function (err, response, result) {
console.log(response.statusCode);
console.log(response.headers);
});ineed.collect.title.from({
url: 'https://test.domain/',
method: 'POST',
form: 'input=test'
}, function (err, response, result) {
console.log(result);
});
Collects information specified by plugin set. The result of the action is an object that contains individual plugin outputs as properties.
Example:
var result = ineed.collect.texts.images.scripts.fromHtml(html);will produce result:
{
"texts" : ,
"images" : ,
"scripts" :
}
…
js
//Delete all HTML comments and render emoji
var result = ineed.reprocess
.comments(function() {
return null;
})
.texts(function(text, escapeHtml) {
return escapeHtml(text).replace(/:beer:/g, '');
})
.fromHtml(html);
…
js
ineed
.using(myPlugin1)
.using(myPlugin2)
.collect
...Plugins of both kinds are objects and in addition to the kind-specific properties they should have the following properties:
Indicates which action will be extended by plugin. Can be 'collect' or 'reprocess'. Required property.
Name of the plugin. Required property. It should reflect the target of the plugin action. Plugin will be accessable under the name in the .collect or .reprocess objects and will be used as result property name for .collect action. E.g. plugin has name='tagNames'. If it extends .collect:
//Enable plugin and use it
var result = ineed.using(plugin).collect.tagNames.fromHtml(html);
//Access plugin results
var pluginResults = result.tagNames;If it extends .reprocess:
var reprocessedHtml = ineed
.using(plugin)
.reprocess
.tagNames(function (tagName) {
if (tagName === 'applet')
return 'object';
})
.fromHtml(html);
…
js
{
name: [String],
publicId: [String],
systemId: [String]
}.onStartTag(startTag)Where startTag:
{
tagName: [String],
attrs: [Array],
selfClosing: [Boolean]
}.onEndTag(tagName).onText(text).onComment(commentText)Collecting plugins in addition to common properties should have .getCollection() method that should return items collected by plugin.
Example of the collecting plugin:
//Collects tagNames in
var plugin = {
extends: 'collect',
name: 'tagNamesInBody',
init: function (ctx) {
this.ctx = ctx;
this.tagNames = [];
},
addTagName: function (tagName) {
if (this.ctx.inBody && this.tagNames.indexOf(tagName) start and end tags
if (tagName === 'noscript')
return null;
//Replace with
if(tagName === 'div')
return 'p';
return tagName;
}).fromHtml(html);
$ npm testIf you have any questions, please feel free to create an issue here on github.
暂无开放 Issues,或尚未同步最近议题。