#1172·webmagic

关于Spider#setScheduler(Scheduler)存在bug

Author: KamiNoYukiCreated Jul 6, 2024Updated Aug 15, 2024

方法代码:

java
public Spider setScheduler(Scheduler updateScheduler) {
        checkIfRunning();
        //此处存在问题,这样会导致直接将内部的scheduler设置为updateScheduler,然后又尝试自己poll自己进行数据迁徙
        SpiderScheduler oldScheduler = this.scheduler;
        //应该修改为下面这样
        //Scheduler oldScheduler = scheduler.getScheduler();
        scheduler.setScheduler(updateScheduler);
        Request request;
        while ((request = oldScheduler.poll(this)) != null) {
            System.out.println("move oldScheduler task to updateScheduler");
            this.scheduler.push(request, this);
        }
        return this;
    }