中止爬行

作者: Radiergummi创建于 2021年5月25日更新于 2021年11月26日

I have looked at #293 and #289, but those issues are slightly different. We have a crawler library based on `node-crawler` that performs computationally intensive crawling tasks and writes to different output sources, depending on the current task. Due to the nature of our tasks, it is crucial for crawlers to be interruptible and pick up work at the same point later on, and write consistent output files at the same time. Think of a JSON output file, for example, to which an array of objects is written, one object per URL. If the crawler stops, a final closing bracket must be written to the output file to ensure the file is valid JSON.

内容来源: bda-research/node-crawler