#437 robots.txt that fails to fetch is cached as allow-all for the whole run

Have you searched if there an existing issue for this?

I have searched the existing issues

Python version (python --version)

Python 3.14.0

Scrapling version (scrapling.version)

0.4.15 (reproduced on dev at 458e2a2)

Dependencies version (pip3 freeze)

anyio==4.15.0
cssselect==1.5.0
curl_cffi==0.16.3
lxml==6.1.3
msgspec==0.21.1
orjson==3.12.0
patchright==1.62.3
playwright==1.62.0
Protego==0.6.2
scrapling==0.4.15

What's your operating system?

macOS 26.6

Are you using a separate virtual environment?

Yes

Expected behavior

With robots_txt_obey = True, a robots.txt that cannot be fetched should not be read as "everything is allowed". RFC 9309 section 2.3.1.3 puts it the other way round: a server error means the crawler should assume complete disallow, at least while the failure lasts. A 404 does mean allow-all (2.3.1.2), and that part is already right.

Actual behavior

RobotsTxtManager._get_parser starts with content = "" and only fills it on status == 200. Every other outcome - 5xx, 429, a connection error, a timeout - falls through to Protego.parse(""), which allows everything, and that parser is then stored in self._cache[domain]. There is no TTL and nothing invalidates it, so one transient failure disables robots.txt for that domain for the rest of the run.

content = ""
try:
    response = await self._fetch_fn(robots_url, sid)
    if response.status == 200:
        content = response.body.decode(response.encoding, errors="replace")
except Exception as e:
    log.warning(f"Failed to fetch robots.txt for {domain}: {e}")

Output of the script below:

[2026-09-04 15:06:37] WARNING: Failed to fetch robots.txt for example.com: timed out
503  -> can_fetch: True
err  -> can_fetch: True
404  -> can_fetch: True
200  -> can_fetch: False

The last two lines are correct. The first two are the bug, and they only bite users who explicitly opted into robots_txt_obey.

A second, smaller thing in the same file: the user agent is hardcoded to "*" in can_fetch and get_delay_directives, so a robots.txt group that names the crawler is never applied. Same script, fifth case:

User-agent: Scrapling
Disallow: /private

User-agent: *
Allow: /

gives can_fetch("https://example.com/private") -> True. I am less sure this one is a bug, since the fetchers randomise the UA and there may be no single name to match, so treating it separately seems right.

I have not opened a PR. The fix for the first part needs a decision I should not make for you: whether a 5xx should be a hard disallow for the whole run, a disallow that is retried later (a cache entry with a TTL, or simply not caching failures), or left as is with a louder warning. Happy to send whichever you prefer, with tests.

Steps To Reproduce

import anyio
from types import SimpleNamespace
from scrapling.spiders.robotstxt import RobotsTxtManager

class Resp(SimpleNamespace): pass

async def main():
    async def fetch_503(url, sid):
        return Resp(status=503, body=b"", encoding="utf-8")
    print("503  -> can_fetch:", await RobotsTxtManager(fetch_503).can_fetch("https://example.com/private", "s"))

    async def fetch_raise(url, sid):
        raise TimeoutError("timed out")
    print("err  -> can_fetch:", await RobotsTxtManager(fetch_raise).can_fetch("https://example.com/private", "s"))

    async def fetch_404(url, sid):
        return Resp(status=404, body=b"", encoding="utf-8")
    print("404  -> can_fetch:", await RobotsTxtManager(fetch_404).can_fetch("https://example.com/private", "s"))

    async def fetch_ok(url, sid):
        return Resp(status=200, body=b"User-agent: *\nDisallow: /private\n", encoding="utf-8")
    print("200  -> can_fetch:", await RobotsTxtManager(fetch_ok).can_fetch("https://example.com/private", "s"))

anyio.run(main)

AI disclosure per AI_POLICY.md: this issue was investigated and written with Claude Code (Claude Opus 5). The reproduction script above was run locally against dev; the output is real.