Cloned Collectors inherit checkRedirectFunc (so it uses the parent's AllowURLRevisit)
Hey Guys, awesome package, I've been using for a while now.
One of my scrapers broke overnight, with the error "already visited", but I had already encountered this on development, and setting it's AllowURLRevisit to true had solved it.
When debugging, I noticed the Collector in checkRedirectFunc had it set to false, and I discovered that it was the parent Collector, from whom I had cloned the one that revisits the same url multiple times.
I took a look at the source code, and the Clone function doesn't update the HttpClient's CheckRedirect function, so that seems to be the issue.
The final nail in the coffin was setting the parent's AllowURLRevisit to true, and seeing the error vanishing again.
Here, the Init function initializing the c.backend.Client.CheckRedirect function:
And the Clone function doesn't update it:
Why it was working before, and why it stopped, I don't know...
Source: gocolly/colly