Feature: Expose JSON path scalar indexes through embedded and remote APIs

Description

Lance can build a scalar index for a path inside a lance.json field using its JSON index wrapper, but LanceDB's public index configuration can only name a top-level column and cannot express the JSON path or target scalar index.

Attempts such as meta.a, meta['a'], or json_get_int(meta, 'a') are resolved as schema field names and fail. Users therefore cannot create the existing Lance JSON-path index through either the embedded or remote LanceDB API.

Proposed API contract

Expose a structured JSON-path index configuration, conceptually:

table.create_index(
    "meta",
    config=JsonPath(path="a", target=BTree()),
)

The exact spelling is open to design. The configuration must keep the source column, path, and target index type separate instead of encoding an expression into a column-name string.

The first increment may support only BTree targets if that keeps the contract small.

Work required

• Add a JSON-path index variant to the LanceDB Rust index API.
• Map embedded creation to Lance's JSON index parameters.
• Expose the configuration in supported SDK bindings.
• Serialize the same structured configuration for remote tables.
• Preserve the configuration in returned index metadata and statistics.
• Document supported path syntax and target index types.

Acceptance criteria

• Embedded and remote clients use the same structured configuration.
• A BTree index can be created for a nested JSON scalar path.
• Equality and range filters on the indexed path return correct results.
• The query plan confirms that the JSON scalar index is used.
• Invalid paths, unsupported targets, and non-JSON source fields fail with actionable errors.
• Embedded integration tests and remote request-serialization tests cover the new configuration.

Upstream dependencies

• JSON support epic: https://github.com/lance-format/lance/issues/4516
• Stable public JSON index parameters: https://github.com/lance-format/lance/issues/5311