[BUG] Mixed-content routing can lose the parent HTML type
Description
Some complete HTML documents are routed as mixed content even though their top-level structure is clearly HTML.
After the document enters the mixed-content path, it is split into smaller sections. These sections are then detected independently, and some of them are classified as plain text or other types instead of HTML.
As a result, the original parent HTML structure is lost and HTMLExtractor does not get a proper chance to process the document.
Observed behavior
The affected inputs were complete HTML/XHTML documents with strong structural indicators such as:
DOCTYPE<html>root element- normal HTML markup structure
However, because the content also contained text, scripts, JSON, or other embedded structures, it entered the mixed-content path.
After splitting, several child sections were no longer treated as HTML.
In one experiment, forcing the split sections to retain the parent HTML type reduced the content from about 157,827 tokens to 810 tokens, roughly a 99.5% reduction.
This suggests that HTMLExtractor itself works well. The problem appears to happen earlier, during content classification and mixed-content routing.
Expected behavior
A clearly identifiable container format should take precedence over generic mixed-content detection.
For example, if the document has a valid HTML root structure, HTML should remain the dominant parent type even if it contains:
- JavaScript
- embedded JSON
- text
- tables
- styles
Mixed content should describe the internal composition of the document, not necessarily replace the top-level document type.
Source: headroomlabs-ai/headroom