Four streaming defects: chunk-boundary UTF-8 corruption, cached line-ending guess, header dedupe on resume, and no backpressure under pause()
Author: lenamonjCreated Jul 27, 2026Updated Sep 9, 2026
Each item below has an offline reproduction. Note your CI is green at this commit, so none of this is visible from the suite as it stands.
- Multi-byte UTF-8 split across a chunk boundary decodes to
U+FFFD(papaparse.js:891). Any non-ASCII file parsed with chunking can silently lose characters. - The line ending is guessed from a truncated first chunk and then cached for the whole file (
papaparse.js:1091). WithchunkSize: 1024on a CRLF file this leaves a stray\ron every last field, with no error raised and a correct row count, so it looks like clean data. - Header dedupe re-runs on every resumed row and rewrites data in place (
papaparse.js:1750). This reproduces issues #985 and #998 verbatim. pause()applies no backpressure (papaparse.js:846-856). 100% of the stream buffers while paused; a 400 MB file under--max-old-space-size=128dies withFATAL ERROR: JavaScript heap out of memory, where the unpaused control completes at 78 MB RSS.
Two things I checked that are not broken, so nobody wastes time on them: quoted fields spanning chunk boundaries are exact at every chunk size across four corpora, and the pause/resume regression test that fails on Windows is a timer-resolution artifact (setTimeout(3) costs ~15.8 ms there, and resume schedules one timer per row), not a deadlock.
I have a patch taking the suite to 248 passing / 0 failing with lint clean. Several existing open PRs (#1099, #909, #1107, #989, #1017, #1131) cover adjacent ground and predate this. Happy to open a PR or leave it here.
Source: mholt/PapaParse