#2977·scrapy

Response data is not released after processing?

Author: netcafCreated Oct 26, 2017Updated Aug 26, 2026
Labelsperformance

Hello,

code sample:

def start_requests(self):
    for url in url_list:
            yield scrapy.Request(url=url, headers = xxx, callback=self.parse, errback=None,dont_filter=False)

def parse(self, response):
    ...
    response.xpath('//*[@id="feed-main-list"]/li')
    ...

If we call selector related function like xpath(...), the response data will not be released after parsing of each url.

In scrapy/http/response/text.py,

class TextResponse(Response):
...
    def selector(self):
        from scrapy.selector import Selector
        if self._cached_selector is None:
            self._cached_selector = Selector(self)
        return self._cached_selector

    def xpath(self, query, **kwargs):
        return self.selector.xpath(query, **kwargs)

It has the cyclic reference like response -> Selector -> response. In my rough test, if i don't use self._cached_selector to keep Selector object, the memory usage will reduce from 6x% -> 4x%.

So, am i right? or how to understand it?