FetchNode passes the page HTML as convert_to_md's baseurl, inflating model input ~74x
Version: scrapegraphai 2.2.2 (call site unchanged on master), Python 3.14, macOS
What happens
FetchNode.handle_web_source calls (scrapegraphai/nodes/fetch_node.py:398 on master):
parsed_content = convert_to_md(document[0].page_content, parsed_content)At that point parsed_content is document[0].page_content, i.e. the full HTML of the page.
convert_to_md(html, url) assigns its second argument to html2text.HTML2Text().baseurl, so
html2text prepends the entire HTML document to every relative link it finds.
Reproduction (no LLM needed)
import urllib.request
from scrapegraphai.utils.convert_to_md import convert_to_md
raw = urllib.request.urlopen("https://books.toscrape.com/").read().decode()
print(len(raw)) # 51274
print(len(convert_to_md(raw, "https://books.toscrape.com/"))) # 13843 <- correct
print(len(convert_to_md(raw, raw))) # 3816199 <- what FetchNode doesEnd to end with SmartScraperGraph on the same page, ParseNode reports 1,948,618 chars of
parsed content instead of ~13.8k. All of it is sent as model input, and the markdown links are
unusable because each href contains the whole document.
With google/gemini-2.5-flash that is roughly 0.15 USD per run instead of 0.001 USD, for an
identical answer. On a page with many relative links the factor is ~74x.
Expected
baseurl should be the source URL, so relative links become absolute.
Suggested fix
parsed_content = convert_to_md(document[0].page_content, source)The use_soup branch (fetch_node.py:302) looks affected too:
parsed_content = convert_to_md(source, parsed_content)Here source is the URL and parsed_content the HTML, so the two arguments appear swapped.
In that same branch parsed_content is unbound when cut is True, because it is only assigned
inside if not self.cut:.
Source: ScrapeGraphAI/Scrapegraph-ai