Add a signal (request_ignored)

Summary

This feature request proposes the addition of a new global signal, for example request_ignored. This signal would be emitted whenever an IgnoreRequest exception is raised or returned from a downloader middleware's process_response method. The signal should provide access to the ignored request, the response that triggered the action, the IgnoreRequest exception instance, and the middleware that triggered it.

Motivation

IgnoreRequest is a powerful flow-control mechanism in Scrapy, but there is currently no direct and clean way to globally monitor which requests are being ignored and why. This capability is crucial for debugging and for operating crawlers in a production environment. The primary use cases include: Debugging Complex Issues: When a spider fails to scrape certain pages, it can be difficult to quickly determine if the issue stems from a middleware (e.g., HttpCompressionMiddleware, RedirectMiddleware,, or a custom one) silently dropping the request. A global signal would make such issues immediately visible. Data Integrity and Auditing: In production, logging all ignored requests (e.g., those that hit max retries or were filtered by a custom rule) is essential for data auditing and analysis. We could store these ignored requests in a database or log file for manual processing or re-crawling later. Advanced Crawling Logic: When a request is ignored, we might want to trigger compensatory actions, such as rotating a proxy or flagging the source URL as problematic. A signal would allow for this kind of decoupled, event-driven logic to be implemented easily. Ultimately, adding this signal would greatly improve Scrapy's observability and provide developers with an officially supported, elegant way to handle ignored requests without having to modify Scrapy's core components.

code like

    @deferred_f_from_coro_f
    async def _scrape(self, result: Response | Failure, request: Request) -> None:
        """Handle the downloaded response or failure through the spider callback/errback."""
        if not isinstance(result, (Response, Failure)):
            raise TypeError(
                f"Incorrect type: expected Response or Failure, got {type(result)}: {result!r}"
            )

        assert self.crawler.spider
        output: Iterable[Any] | AsyncIterator[Any]
        if isinstance(result, Response):
            try:
                # call the spider middlewares and the request callback with the response
                output = await self.spidermw.scrape_response_async(
                    self.call_spider, result, request, self.crawler.spider
                )
            except Exception:
                self.handle_spider_error(Failure(), request, result)
            else:
                await self.handle_spider_output_async(output, request, result)
            return

        try:
            # call the request errback with the downloader error
            output = await self.call_spider_async(result, request)
        except Exception as spider_exc:
            # the errback didn't silence the exception
            if not result.check(IgnoreRequest):
                logkws = self.logformatter.download_error(
                    result, request, self.crawler.spider
                )
                logger.log(
                    *logformatter_adapter(logkws),
                    extra={"spider": self.crawler.spider},
                    exc_info=failure_to_exc_info(result),
                )
            else:
                # add this code
                await self.signals.send_catch_log_async(
                    signal=signals.request_ignored,
                    request=request,
                    exception=result,
                    spider=self.crawler.spider
                    )

            if spider_exc is not result.value:
                # the errback raised a different exception, handle it
                self.handle_spider_error(Failure(), request, result)
        else:
            await self.handle_spider_output_async(output, request, result)

Describe alternatives you've considered

Clone a scrapy code, develop locally and then publish it to pypi, but this will result in the inability to use the latest version of scrapy