Response data is not released after processing?
Author: netcafCreated Oct 26, 2017Updated Aug 26, 2026
Labelsperformance
Hello,
code sample:
def start_requests(self):
for url in url_list:
yield scrapy.Request(url=url, headers = xxx, callback=self.parse, errback=None,dont_filter=False)
def parse(self, response):
...
response.xpath('//*[@id="feed-main-list"]/li')
...
If we call selector related function like xpath(...), the response data will not be released after parsing of each url.
In scrapy/http/response/text.py,
class TextResponse(Response):
...
def selector(self):
from scrapy.selector import Selector
if self._cached_selector is None:
self._cached_selector = Selector(self)
return self._cached_selector
def xpath(self, query, **kwargs):
return self.selector.xpath(query, **kwargs)
It has the cyclic reference like response -> Selector -> response. In my rough test, if i don't use self._cached_selector to keep Selector object, the memory usage will reduce from 6x% -> 4x%.
So, am i right? or how to understand it?
Source: scrapy/scrapy