百科.dev
全部条目在线工具AI 编程趋势榜开源项目技术资讯提交条目
登录
< 返回工具列表
C

crawler

> 编程语言
开源

用于快速 (Web) 爬虫和抓取器开发的库

369 stars0 点赞1 次浏览
访问官网GitHub

工具介绍

用于快速 (Web) 爬虫和抓取器开发的库

Library for Rapid (Web) Crawler and Scraper Development

This library provides kind of a framework and a lot of ready to use, so-called steps, that you can use as building blocks, to build your own crawlers and scrapers with.

To give you an overview, here's a list of things that it helps you with:

  • Crawler Politeness 😇 (respecting robots.txt, throttling,...)
  • Load URLs using
    • a (PSR-18) HTTP client (default is of course Guzzle)
    • or a headless browser (chrome) to get source after Javascript execution
  • Get absolute links from HTML documents 🔗
  • Get sitemaps from robots.txt and get all URLs from those sitemaps
  • Crawl (load) all pages of a website 🕷
  • Use cookies (or don't) 🍪
  • Use any HTTP methods (GET, POST,...) and send any headers or body
  • Easily iterate over paginated list pages 🔁
  • Extract data from:
    • HTML and also XML (using CSS selectors or XPath queries)
    • JSON (using dot notation)
    • CSV (map columns)
  • Extract schema.org structured data in JSON-LD format from HTML documents
  • Keep memory usage low by using PHP Generators 💪
  • Cache HTTP responses during development, so you don't have to load pages again and again after every code change
  • Get logs about what your crawler is doing (accepts any PSR-3 LoggerInterface)
  • And a lot more...

Documentation

You can find the documentation at crwlr.software.

Contributing

If you consider contributing something to this package, read the contribution guide (CONTRIBUTING.md).

Issues· 0 开放

查看全部 Issues在 GitHub 打开

暂无开放 Issues,或尚未同步最近议题。

> 标签

PHPcrawlercrawlinghacktoberfestphp

暂无评论,来聊聊你的看法吧

> 工具信息

发布日期2026年8月1日
最后更新2026年9月17日
分类编程语言
定价开源

> 相关工具

T
TypeScript
JavaScript 的超集,为前端与全栈提供静态类型
P
Python
通用编程语言,广泛用于 Web、数据与 AI
G
Go
Google 推出的简洁高效系统语言