gecco

Open source

Easy to use lightweight web crawler（易用的轻量化网络爬虫）

2.5K stars0 likes0 views

About

What is Gecco

Gecco is a easy to use lightweight web crawler developed with java language.Gecco integriert jsoup, httpclient, fastjson, spring, htmlunit, redission ausgezeichneten framework,Let you only need to configure a number of jQuery style selector can be very quick to write a crawler.Gecco framework has excellent scalability, the framework based on the principle of open and close design, to modify the closure, the expansion of open.At the same time Gecco is based on a very open MIT open source protocol, whether you are a user or want to jointly improve the Gecco developer, welcome to request pull.If you like the crawler framework,please star or fork!

中文说明

Main features

Easy to use, use jQuery style selector to extract elements
Support for asynchronous Ajax requests in the page
Support page JavaScript variable extraction
Using Redis to realize distributed crawling,reference gecco-redis
Support the development of business logic with Spring,reference gecco-spring
Support htmlunit extension,reference gecco-htmlunit
Support extension mechanism
Support download UserAgent random selection
Support the download proxy server randomly selected

Framework overview

Download

Download via Maven

xml

<dependency>
    <groupId>com.geccocrawler</groupId>
    <artifactId>gecco</artifactId>
    <version>x.x.x</version>
</dependency>

Dependent project

httpclient，jsoup，fastjson，reflections，cglib，rhino，log4j，jmxutils，commons-lang3

Quick start

…

DynamicGecco

The purpose of DynamicGecco is to implement the runtime configuration of the crawl rule without defining the SpiderBean.In fact, the principle is the use of byte code programming, dynamic generation of SpiderBean, but also through the custom GeccoClassLoader to achieve the rule of hot deployment.Below is a simple Demo, more complex Demo can refer to the example below com.geccocrawler.gecco.demo.dynamic.

The following code implements the runtime configuration of the crawl rule:

DynamicGecco.html()
.gecco("https://github.com/{user}/{project}", "consolePipeline")
.requestField("request").request().build()
.stringField("user").requestParameter("user").build()
.stringField("project").requestParameter().build()
.stringField("star").csspath(".pagehead-actions li:nth-child(2) .social-count").text(false).build()
.stringField("fork").csspath(".pagehead-actions li:nth-child(3) .social-count").text().build()
.stringField("contributors").csspath("ul.numbers-summary > li:nth-child(4) > a").href().build()
.register();

GeccoEngine.create()
.classpath("com.geccocrawler.gecco.demo")
.start("https://github.com/xtuhcy/gecco")
.run();

You can see that the DynamicGecco way compared to the traditional way of annotation code greatly reduced, and a very cool point is DynamicGecco to support the operation of the definition and modification of rules.