bug: schema validation misses nested duplicate names and reserved system column names on several write paths

Schema::validate() enforces two of the invariants we rely on — unique top-level field names and unique field ids — but it misses sibling names inside structs, and it does not enforce the reserved-system-column rule at all. The system-column rule is instead spot-checked at a handful of write call sites, and several write paths are not among them.

lance_core::is_system_column (rust/lance-core/src/lib.rs:66) already states the contract in its doc comment:

Write paths must reject a stored column named for one: the scanner injects these itself, so a stored copy collides with the injected one on read.

That contract is currently enforced by convention, not by construction.

What is enforced today

Schema::validate() (rust/lance-core/src/datatypes/schema.rs:330) rejects:

• . in top-level field names
• duplicate top-level field names
• negative and duplicate field ids (recursive, via fields_pre_order())
• zero-dimension fixed-size lists

Schema::try_from(&ArrowSchema) (schema.rs:892) calls it, so every Arrow→Lance conversion gets those checks.

The reserved-name rule is checked separately at:

• InsertBuilder::validate_write — rust/lance/src/dataset/write/insert.rs:340
• FileFragment column-data staging — rust/lance/src/dataset/fragment.rs:2625

Gaps

1. Duplicate sibling names inside a struct are accepted

The name-uniqueness loop in validate() iterates self.fields (top level only); only the id check walks fields_pre_order(). So st: struct<x: int32, x: int64> validates.

Verified by running a probe against lance-core:

dup top:    Err(Schema { message: "Duplicate field name \"x\" in schema: ..." })
dup nested: Ok("ACCEPTED")

project(&["st.x"]), Field::do_intersection and Schema::merge all resolve children by name, so the second x is unreachable and which one wins is positional.

2. Reserved system column names are accepted at the schema level

Also verified by probe:

system-col top-level (_rowid): Ok("ACCEPTED")
system nested (st.{_rowid}):   Ok("ACCEPTED")

Nested is a gap even for the two guarded call sites above — both iterate top-level fields only.

3. Write paths with no reserved-name guard

These are from reading the code, not executed:

The symptom in each case is a stored column that collides with the scanner-injected virtual column on read — the same class of failure as #4358, but reached through a write path rather than a query.

Suggested direction

Sibling-name uniqueness can move into Schema::validate() unconditionally, checking uniqueness among siblings at every level rather than only at the root. No read path depends on duplicate sibling names being legal. (Worth keeping an eye on the cost here given #8273 — the fix should stay a per-parent hash set, not a quadratic scan.)

The reserved-name rule cannot go into validate() unconditionally, because validate() also runs on read paths: Schema is reconstructed from the manifest protobuf, and project_preserve_system_columns deliberately builds schemas containing ROW_ID_FIELD (schema.rs:280, schema.rs:1525). Options:

1. A separate recursive validate_writable() (or similar) called from a single chokepoint on the write side — Transaction construction or CommitBuilder. This catches every writer including the distributed one, and it is the only place that genuinely knows "this schema is about to become the manifest."
2. A flag on validate().
3. Keep projection schemas out of Schema entirely.

Option 1 seems right: it replaces five-plus ad hoc call sites with one guard at the boundary, and it is the only option that closes the FragmentCreateBuilder + CommitBuilder path without asking every future writer to remember the rule.

Happy to be told the chokepoint belongs somewhere else — the placement is the part worth arguing about.

Tests worth adding

• nested duplicate sibling names rejected by Schema::validate()
• _rowid (and each of the other four system names) rejected at top level and nested
• add_columns with a system column name rejected
• alter_columns renaming an existing column to a system column name rejected
• write_fragments + CommitBuilder with a system column in the schema rejected