Fetch llms.txt
Author: DavideDelbiancoCreated Sep 17, 2025Updated Sep 17, 2025
It would be nice if you attempted to fetch llms.txt the same way you fetch robots.txt It is still a proposal, but more and more sites are starting to adopt it.
Source: jina-ai/reader