HTML backend: non-<li> children of <ul>/<ol> (e.g. <p>, <table>) are silently dropped
Bug
The HTML backend silently drops content that is a direct child of <ul>/<ol> but is not an <li>. Such markup is invalid HTML, but CMS output contains it regularly. Browsers render the content, and BeautifulSoup keeps it in the parse tree, but it is missing from the converted document. There is no warning.
Input:
<ul>
<li>First item</li>
<p>A paragraph placed directly inside the list.</p>
<li>Second item</li>
</ul>
<p>After.</p>Actual:
list_item: First item
list_item: Second item
text: After.Expected: the paragraph is kept in document order. Browsers render it inside the list, below the preceding item, so it belongs to that list item (a ListGroup can only contain ListItem children):
list_item: First item
text: A paragraph placed directly inside the list. (child of the list item)
list_item: Second item
text: After.A <table> placed directly inside a list is dropped the same way. A <ul>/<ol> placed directly inside a list is not dropped, but it is added to the parent ListGroup as a group, and the numbering of the following items is off:
<ol><li>a</li><li>b</li><ol><li>x</li></ol><li>c</li></ol>1. a
2. b
1. x
4. cExamples in published pages:
- https://lovdata.no/nav/rundskriv/r11-00: each
<li>names a provision and is followed by a<p>sibling that explains it. All 7 paragraphs are missing from the output. - https://www.finanstilsynet.no/nyhetsarkiv/rundskriv/2022/veileder-til-hvitvaskingsloven/ and https://www.arbeidstilsynet.no/lonn-og-ansettelse/ansettelse/innleie-av-avloser/:
<ul>placed directly inside<ul>.
Steps to reproduce
from io import BytesIO
from docling.datamodel.base_models import DocumentStream, InputFormat
from docling.document_converter import DocumentConverter
html = """<!DOCTYPE html><html><head><meta charset="utf-8"></head><body>
<ul><li>First item</li><p>A paragraph placed directly inside the list.</p><li>Second item</li></ul>
<p>After.</p>
</body></html>"""
converter = DocumentConverter(allowed_formats=[InputFormat.HTML])
document = converter.convert(DocumentStream(name="list_children.html", stream=BytesIO(html.encode()))).document
for item, _ in document.iterate_items():
print(f"{item.label.value}: {getattr(item, 'text', '')}")Docling version
2.130.0 (also reproduced with 2.124.0); docling-core 2.99.0; beautifulsoup4 4.15.0
Python version
Python 3.13.9
Source: docling-project/docling