#875·colly

Cloned Collectors inherit checkRedirectFunc (so it uses the parent's AllowURLRevisit)

Author: gabrielbergocCreated May 29, 2026Updated May 29, 2026

Hey Guys, awesome package, I've been using for a while now.

One of my scrapers broke overnight, with the error "already visited", but I had already encountered this on development, and setting it's AllowURLRevisit to true had solved it. When debugging, I noticed the Collector in checkRedirectFunc had it set to false, and I discovered that it was the parent Collector, from whom I had cloned the one that revisits the same url multiple times. I took a look at the source code, and the Clone function doesn't update the HttpClient's CheckRedirect function, so that seems to be the issue. The final nail in the coffin was setting the parent's AllowURLRevisit to true, and seeing the error vanishing again.

Here, the Init function initializing the c.backend.Client.CheckRedirect function: Image

And the Clone function doesn't update it: Image

Why it was working before, and why it stopped, I don't know...