read_ndjson / scan_ndjson silently drop a key first seen past infer_schema_length, while read_json raises on the same records

Checks

• I have checked that this issue has not already been reported.
• I have confirmed this bug exists on the latest version of Polars.

Reproducible example

import io, json, polars as pl

# 1000 NDJSON records of {"a": i}; an optional second key "b" appears from record k onwards.
def make(k):
    return "\n".join(
        json.dumps({"a": i} if i < k else {"a": i, "b": i * 10}) for i in range(1000)
    ).encode()

for k in (99, 100):
    df = pl.read_ndjson(make(k))
    got = int(df["b"].is_not_null().sum()) if "b" in df.columns else 0
    print(f"first record carrying 'b' = {k:3d} -> columns={df.columns}  values of b recovered={got}")

# first record carrying 'b' =  99 -> columns=['a', 'b']  values of b recovered=901
# first record carrying 'b' = 100 -> columns=['a']       values of b recovered=0     <-- 900 values gone

One record shifts the key's first appearance from 99 to 100 and 900 values disappear. No error, no warning. scan_ndjson(...).collect() gives the same answer, so it is not specific to the eager path.

The same records, handed to read_json as a JSON array, raise instead:

recs = [{"a": i} if i < 100 else {"a": i, "b": i * 10} for i in range(1000)]
pl.read_json(io.BytesIO(json.dumps(recs).encode()))
# ComputeError: extra field in struct data: b, consider increasing infer_schema_length,
#               or manually specifying the full schema to ignore extra fields

Raising the knob recovers everything from the same bytes, which pins this to the sample rather than to parsing:

infer_schema_length = 100   -> columns ['a'],      0 values of b
infer_schema_length = 1000  -> columns ['a','b'], 900 values of b
infer_schema_length = None  -> columns ['a','b'], 900 values of b
schema={"a": pl.Int64, "b": pl.Int64} -> columns ['a','b'], 900 values of b

Log output

Nothing is emitted. The call returns a `DataFrame` with the column absent and raises no
warning; `POLARS_VERBOSE=1` adds nothing relevant.

Issue description

read_ndjson / scan_ndjson infer the schema from the first infer_schema_length records (default 100) and then discard, without any diagnostic, every key whose first occurrence is later in the file. The boundary is exact — record 99 is fine, record 100 is not — so this is not a size- or cost-based heuristic.

Two things make this a defect rather than a documented trade-off:

1. Polars already has the error, the message and the advice for this exact condition — the NDJSON reader just doesn't use them. read_json, on the same records, stops and names the knob to turn (extra field in struct data: b, consider increasing infer_schema_length, or manually specifying the full schema to ignore extra fields). One reader in the same library, same records, same condition, reports it; the sibling returns a truncated frame in silence.

2. read_ndjson itself raises for every other post-sample surprise. With the column sampled as Int64 from the first 100 records and a conflicting value at record 500:

The machinery to notice a post-sample surprise exists and runs on every record. It fires for six shapes out of seven; the case it misses is the one that loses the most data.

The documentation does not cover this either. The infer_schema_length docstring reads "The maximum number of rows to scan for schema inference. If set to None, the full data may be scanned (this is slow)." — a performance trade-off, with no indication that a lower value silently drops fields.

It applies one level down as well. With {"a": {"x": i}} and a nested y appearing from record k:

k =  99  ->  dtype Struct({'x': Int64, 'y': Int64})   901 values of a.y
k = 100  ->  dtype Struct({'x': Int64})                 0 values of a.y

Why the ordering matters. Whether a field survives depends on where in the file it first appears, not on the file's contents. Two NDJSON files holding exactly the same multiset of records give different answers depending on the order they were written in. Any export where an optional field happens to first appear on line 100 comes back without it.

Suggested resolution, in preference order:

1. Make read_ndjson / scan_ndjson behave like read_json and raise the existing extra field in struct data: … error, which already names the fix.
2. Failing that, emit a warning naming the dropped key(s).

There is precedent for treating this class as a real bug rather than a documented limit: #13061 ("Silent data loss in json_decode with default infer_schema_length", bug/accepted/P-high) was resolved by making dtype a required argument of str.json_decode, removing the silent inference path entirely. This is the same failure mode in a different reader, and the sibling reader already demonstrates the desired behaviour.

Related but distinct: #23190 (open) reports the same underlying mechanism through pl.DataFrame(list_of_dicts) — I confirmed it still reproduces on 1.44.2 — and argues for raising the default. This report is narrower and does not depend on that: even keeping the default at 100, the NDJSON reader should not be the only JSON entry point that stays silent.

Expected behavior

Either of:

pl.read_ndjson(make(100))
# ComputeError: extra field in struct data: b, consider increasing infer_schema_length,
#               or manually specifying the full schema to ignore extra fields

matching read_json on the same records, or a frame containing b with its 900 values. What should not happen is a frame that is missing the column with no diagnostic of any kind.

Installed versions

--------Version info---------
Polars:              1.44.2
Index type:          UInt32
Platform:            Linux-5.15.0-187-generic-x86_64-with-glibc2.35
Python:              3.10.12 (main, Jul 15 2026, 23:40:17) [GCC 11.4.0]
Runtime:             rt32

----Optional dependencies----
numpy                2.2.6
matplotlib           3.10.9
(all others not installed)

Also reproduced on 2.0.0rc1, and — per the folder's earlier sweep — on 0.20.31, 1.0.0, 1.10.0, 1.20.0, 1.30.0 and 1.40.0, so this is longstanding rather than a regression.