FetchNode passes the page HTML as convert_to_md's baseurl, inflating model input ~74x

Author: julianrieglerCreated Sep 1, 2026Updated Sep 3, 2026

Version: scrapegraphai 2.2.2 (call site unchanged on master), Python 3.14, macOS

What happens

FetchNode.handle_web_source calls (scrapegraphai/nodes/fetch_node.py:398 on master):

python
parsed_content = convert_to_md(document[0].page_content, parsed_content)

At that point parsed_content is document[0].page_content, i.e. the full HTML of the page. convert_to_md(html, url) assigns its second argument to html2text.HTML2Text().baseurl, so html2text prepends the entire HTML document to every relative link it finds.

Reproduction (no LLM needed)

python
import urllib.request
from scrapegraphai.utils.convert_to_md import convert_to_md

raw = urllib.request.urlopen("https://books.toscrape.com/").read().decode()
print(len(raw))                                                # 51274
print(len(convert_to_md(raw, "https://books.toscrape.com/")))  # 13843    <- correct
print(len(convert_to_md(raw, raw)))                            # 3816199  <- what FetchNode does

End to end with SmartScraperGraph on the same page, ParseNode reports 1,948,618 chars of parsed content instead of ~13.8k. All of it is sent as model input, and the markdown links are unusable because each href contains the whole document.

With google/gemini-2.5-flash that is roughly 0.15 USD per run instead of 0.001 USD, for an identical answer. On a page with many relative links the factor is ~74x.

Expected

baseurl should be the source URL, so relative links become absolute.

Suggested fix

python
parsed_content = convert_to_md(document[0].page_content, source)

The use_soup branch (fetch_node.py:302) looks affected too:

python
parsed_content = convert_to_md(source, parsed_content)

Here source is the URL and parsed_content the HTML, so the two arguments appear swapped. In that same branch parsed_content is unbound when cut is True, because it is only assigned inside if not self.cut:.

Source: ScrapeGraphAI/Scrapegraph-ai