Reddit listing pages are parsed as a single post, silently dropping the rest of the feed
Problem
On a Reddit listing page such as https://www.reddit.com/, the synchronous extractor returns the body of the first post and silently omits the other loaded posts. This makes feed summaries incomplete even when the content is already present in the DOM.
Tested with Defuddle 0.19.3, using both browser DOM extraction in Chrome and the standalone reproduction below. No authentication, network requests, shadow DOM, or extension-specific preprocessing are needed for the minimal reproduction.
Minimal reproduction
Install [email protected] and [email protected], then run this as an ES module with Node.js 22.12+:
import { JSDOM } from 'jsdom';
import { Defuddle } from 'defuddle/node';
const url = 'https://www.reddit.com/';
const dom = new JSDOM(`<!doctype html>
<title>Reddit feed</title>
<main>
<shreddit-post post-title="First post" author="example_a">
<div slot="text-body"><p>FIRST_POST_SENTINEL: First post body.</p></div>
</shreddit-post>
<shreddit-post post-title="Second post" author="example_b">
<div slot="text-body"><p>SECOND_POST_SENTINEL: Second post body.</p></div>
</shreddit-post>
</main>`, { url });
try {
const result = await Defuddle(dom.window.document, url, {
markdown: true,
useAsync: false,
});
console.log(result.content);
const text = result.content.replace(/\\/g, '');
console.log({
firstPost: text.includes('FIRST_POST_SENTINEL'),
secondPost: text.includes('SECOND_POST_SENTINEL'),
});
} finally {
dom.window.close();
}Actual result:
{ firstPost: true, secondPost: false }Expected: both post bodies are retained when extracting the listing, or the single-post extractor declines this page so the generic pipeline can handle it.
Likely cause
In RedditExtractor, the constructor selects the first shreddit-post, and canExtract() accepts any page containing such an element (or an old-Reddit listing item). It does not distinguish a listing from an individual post page. The extraction then uses that first post's body.
The same applicability check is present in the current main branch at the time of filing. The existing comments-page check is used for async extraction, but does not guard synchronous canExtract().
Setting contentSelector: 'main' or relaxing the general cleanup options does not resolve this, because the site-specific extractor is selected before the generic content-selection pipeline.
Browser cross-check
On a loaded Reddit homepage, an experimental local bypass of the site-specific extractor allowed the generic pipeline, with less aggressive cleanup, to retain all 28 checked post links and titles. This is supporting evidence that the immediate problem is extractor selection, rather than unavailable page content; it is not a claim that the generic default settings preserve every feed detail.
Would it make sense to restrict the single-post extractor to individual post pages and let listing pages fall through to the generic pipeline? A regression test with multiple posts on a listing, alongside existing single-post/comment tests, could protect this behavior.
Source: kepano/defuddle