[Persistence] [v2] register_custom_data_class silently requires an undocumented `_schema` attribute; write succeeds, query fails with a Rust-only remedy

Bug Report

Confirmation

• I've re-read the relevant sections of the documentation.
• I've searched existing issues and discussions to avoid duplicates.
• I've reviewed or skimmed the source code (or examples) to confirm the behavior is not by design.
• I've tested this issue using a recent pre-release or development wheel (2.0.0rcN, dev develop, or a nightly) and can still reproduce it.

Reproduced on 2.0.0rc4 (PyPI) and on the development wheel 2.0.0rc5.dev20260912 (packages.nautechsystems.io). Python 3.14.2, pyarrow 24.0.0, macOS arm64.

I searched open and closed issues for register_custom_data_class, query_custom_data, write_custom_data, ensure_custom_data_registered, and the exact error text. Related but distinct: #4607 (Feather-to-Parquet conversion of custom data), #4297 (adapter-native Rust types lacking Arrow registration, closed), #3923 (dynamic Parquet queries missing registered types, closed). The open RFC #4971 / PR #4959 mention that a Python customdataclass will supply the Arrow schema — this report may be resolved by that work, in which case it can serve as a regression case for it.

Summary

register_custom_data_class documents a required contract for a Python custom-data class (type_name_static()/__name__, from_json, decode_record_batch_py, instances with ts_event/ts_init/encode_record_batch_py). A class that satisfies exactly that contract registers successfully and writes successfully, but cannot be queried back:

OSError: Failed to query custom data: custom data type 'SensorReading' is not registered
with an Arrow schema containing ts_init; call ensure_custom_data_registered::<T>() before querying

The query path additionally requires an Arrow schema that, from Python, is only picked up from an undocumented _schema class attribute. Setting _schema = pyarrow.schema([...]) on the class makes query_custom_data return the rows. Nothing in the docstring, the .pyi stubs, or the package's Python sources mentions _schema, and the error message names a Rust generic (ensure_custom_data_registered::<T>()) that a Python caller cannot invoke.

In 1.x the customdataclass_pyo3 decorator set _schema from the class annotations, so a decorated class round-tripped. That decorator is not present in 2.0.0rc4, and no replacement sets the attribute, so a class written to the documented contract is write-only.

Expected behavior

One of:

• A class satisfying the documented contract in register_custom_data_class.__doc__ can be written and queried back, or
• registration (or at the latest write_custom_data) fails immediately with a message that names the missing piece — rather than succeeding and leaving the failure to the first query.

Actual behavior

nautilus_trader 2.0.0rc4
write_custom_data -> data/custom/SensorReading/2023-11-14T22-13-20-000000000Z_2023-11-14T22-13-20-000000002Z.parquet
list_data_types  -> ['custom']
get_intervals    -> [(1700000000000000000, 1700000000000000002)]
query_custom_data ->  OSError: Failed to query custom data: custom data type 'SensorReading' is not registered with an Arrow schema containing ts_init; call ensure_custom_data_registered::<T>() before querying

Identical output on 2.0.0rc5.dev20260912. Adding the _schema attribute (see below) changes the last line to query_custom_data -> 3 rows on both.

Steps to reproduce

Self-contained; no data files needed.

"""A Python-registered custom data class writes to the catalog but cannot be queried back."""
import json, sys, tempfile
import pyarrow as pa
import nautilus_trader
from nautilus_trader.model import CustomData, DataType, register_custom_data_class
from nautilus_trader.persistence import ParquetDataCatalog


class SensorReading:
    # Uncomment the next two lines and query_custom_data succeeds:
    # _schema = pa.schema([("sensor_id", pa.string()), ("value", pa.float64()),
    #                      ("ts_event", pa.uint64()), ("ts_init", pa.uint64())])

    def __init__(self, sensor_id: str, value: float, ts_event: int, ts_init: int):
        self.sensor_id, self.value = sensor_id, value
        self.ts_event, self.ts_init = ts_event, ts_init

    @classmethod
    def type_name_static(cls):
        return "SensorReading"

    def to_json(self):
        return json.dumps(self.__dict__)

    @classmethod
    def from_json(cls, data):
        d = json.loads(data) if isinstance(data, (str, bytes)) else data
        return cls(d["sensor_id"], d["value"], d["ts_event"], d["ts_init"])

    def encode_record_batch_py(self, items):
        return pa.RecordBatch.from_pydict({
            "sensor_id": [i.sensor_id for i in items],
            "value": [i.value for i in items],
            "ts_event": pa.array([i.ts_event for i in items], pa.uint64()),
            "ts_init": pa.array([i.ts_init for i in items], pa.uint64()),
        })

    @classmethod
    def decode_record_batch_py(cls, metadata, batch):
        d = batch.to_pydict()
        return [cls(*row) for row in zip(d["sensor_id"], d["value"], d["ts_event"], d["ts_init"])]


register_custom_data_class(SensorReading)

catalog = ParquetDataCatalog(tempfile.mkdtemp())
T = 1_700_000_000_000_000_000
items = [CustomData(data_type=DataType("SensorReading"),
                    data=SensorReading("s1", float(i), T + i, T + i)) for i in range(3)]

path = catalog.write_custom_data(items)
print("nautilus_trader", nautilus_trader.__version__)
print("write_custom_data ->", path.split("/data/")[-1])
print("list_data_types  ->", catalog.list_data_types())
print("get_intervals    ->", catalog.get_intervals("custom", "SensorReading"))

try:
    rows = catalog.query_custom_data("SensorReading")
    print("query_custom_data ->", len(rows), "rows")
except Exception as e:
    print("query_custom_data -> ", type(e).__name__ + ":", e)
    sys.exit(1)

Run: pip install "nautilus_trader==2.0.0rc4" "pyarrow==24.0.0" && python repro.py → exit 1 with the error above. Uncomment _schema → exit 0, query_custom_data -> 3 rows.

Notes from the reproduction that may help locate the cause:

• decode_record_batch_py is never invoked; the failure occurs before decoding.
• query_files, list_parquet_files, get_intervals, and query_last_timestamp all work for the type (metadata-only paths), so the file itself is well-formed; only the typed read path is affected.
• The same _schema attribute is what the 1.x customdataclass_pyo3 decorator populated, and its encoder read self.__class__._schema with the message "Register the type with register_custom_data_class(...)" — so the attribute appears to have been the intended bridge, and it is currently the only Python-side way to reach the query path.

Possible solutions

Any one of these would resolve it; the first two are small.

1. Document the schema requirement. Add _schema (or, better, a public classmethod such as arrow_schema() -> pyarrow.Schema) to the contract in register_custom_data_class.__doc__ and the .pyi stub, with a note that it must contain ts_init.
2. Fail early with an actionable message. If no Arrow schema is available at registration (or at the latest at write_custom_data), raise there: "custom data type 'X' has no Arrow schema; define _schema/arrow_schema() on the class" — instead of writing successfully and failing on the first query. The current message's remedy (ensure_custom_data_registered::<T>()) is not callable from Python.
3. Derive the schema at write time. encode_record_batch_py returns a RecordBatch that already carries its schema; registering that schema on the first successful write would make any class that can write also readable, with no new contract surface.

Happy to turn (1)/(2) into a PR if that would help, and to re-test against a branch.