SpiderKeeper

SpiderKeeper

A scalable admin ui for spider service

Features

• Manage your spiders from a dashboard. Schedule them to run automatically
• With a single click deploy the scrapy project
• Show spider running stats
• Provide api

Current Support spider service

• Scrapy ( with scrapyd)

Screenshot

Getting Started

Installing

pip install spiderkeeper

Deployment

…

Usage

Visit: 

- web ui : http://localhost:5000

1. Create Project

2. Use [scrapyd-client](https://github.com/scrapy/scrapyd-client) to generate egg file 

   scrapyd-deploy --build-egg output.egg

2. upload egg file (make sure you started scrapyd server)

3. Done & Enjoy it

- api swagger: http://localhost:5000/api.html

TODO

• Job dashboard support filter
• User Authentication
• Collect & Show scrapy crawl stats
• Optimize load balancing

Versioning

We use SemVer for versioning. For the versions available, see the tags on this repository.

Authors

• Initial work - DormyMo

See also the list of contributors who participated in this project.

License

This project is licensed under the MIT License - see the LICENSE.md file for details

Contributing

Contributions are welcomed!

交流反馈

QQ群：

1群: 389688974（已满）

2群: 285668943

捐赠