webmagic

Readme in Chinese

A scalable crawler framework. It covers the whole lifecycle of crawler: downloading, url management, content extraction and persistent. It can simplify the development of a specific crawler.

Features:

• Simple core with high flexibility.
• Simple API for html extracting.
• Annotation with POJO to customize a crawler, no configuration.
• Multi-thread and Distribution support.
• Easy to be integrated.

Install:

Add dependencies to your pom.xml:

<dependency>
    <groupId>us.codecraft</groupId>
    <artifactId>webmagic-core</artifactId>
    <version>${webmagic.version}</version>
</dependency>
<dependency>
    <groupId>us.codecraft</groupId>
    <artifactId>webmagic-extension</artifactId>
    <version>${webmagic.version}</version>
</dependency>

WebMagic use slf4j with slf4j-log4j12 implementation. If you customized your slf4j implementation, please exclude slf4j-log4j12.

<exclusions>
    <exclusion>
        <groupId>org.slf4j</groupId>
        <artifactId>slf4j-log4j12</artifactId>
    </exclusion>
</exclusions>

Get Started:

First crawler:

Write a class implements PageProcessor. For example, I wrote a crawler of github repository information.

…

• page.addTargetRequests(links)

  Add urls for crawling.

You can also use annotation way:

…

Docs and samples:

Documents: http://webmagic.io/docs/

The architecture of webmagic (referred to Scrapy)

There are more examples in webmagic-samples package.

License:

Licensed under Apache 2.0 license

Thanks:

To write webmagic, I refered to the projects below :

• Scrapy

  A crawler framework in Python.

  http://scrapy.org/

• Spiderman

  Another crawler framework in Java.

  http://git.oschina.net/l-weiwei/spiderman

Mail-list:

https://groups.google.com/forum/#!forum/webmagic-java

http://list.qq.com/cgi-bin/qf_invite?id=023a01f505246785f77c5a5a9aff4e57ab20fcdde871e988

QQ Group: 373225642 542327088

Related Project

• Gather Platform

  A web console based on WebMagic for Spider configuration and management.