[RFC] Catalog and Arrow schema v2

Proposal

Introduce a shared catalog API and a revised Arrow storage format for NautilusTrader. This gives catalog consumers a common interface, makes persisted data easier to use with external analytics tools, and extends typed batching to the persistence boundary.

This proposal builds on Typed data transport and heterogeneous boundaries, which establishes the typed batching direction and explicitly leaves catalog APIs, persistent schemas, and depth-model changes to separate proposals.

The open Refactor Parquet catalogs and Arrow encoding PR provides the implementation under discussion, including a migration command and an initial migration guide. This RFC sets out the intended contract, compatibility changes, and requirements for releasing the transition.

“Catalog and Arrow schema v2” names the catalog interface and storage-format generation described here. It does not imply that existing files already carry an explicit version marker.

Thank you to @faysou for the sustained work researching and developing this over a long period of time. The implementation brings together substantial work across storage, serialization, custom data, and runtime integration.

Why make this transition?

The existing catalog format requires Nautilus-specific decoding for several common values. Prices and quantities use fixed-width binary representations tied to the build configuration, and fixed-depth book schemas constrain how snapshots are represented.

Catalog consumers also depend on concrete Parquet interfaces. Supporting another backend therefore involves more than implementing storage operations: callers need a shared definition of queries, writes, coverage, and lifecycle behavior.

The proposed design provides:

• More accessible stored data. Standard Arrow decimals, timestamps, and named enums expose values directly to analytics tools.
• Consistent persisted numeric schemas. A common decimal representation separates storage from the model’s build-dependent raw representation. Readers still reject values their numeric configuration cannot represent exactly.
• A shared catalog boundary. Backtest and live consumers can use the same contracts, while backend-specific behavior remains in the persistence layer.
• Typed query results. Catalogs can return homogeneous batches without requiring every operation to materialize individual Data values.
• Variable-depth snapshots. Storage can retain the available book levels and supplied order IDs.
• Optional Parquet streaming. Captured data can move from Feather staging into a queryable catalog through an explicit promotion lifecycle.

These are architectural and interoperability benefits. Throughput and memory improvements require separate measurements; this proposal does not assume an end-to-end performance gain.

Catalog abstractions

Reading, writing, and backend selection

The catalog interface separates capabilities so consumers can depend on the operations they need.

Abstraction	Responsibility
CatalogReader	Queries, query sessions, identifiers, metadata, and coverage
CatalogWriter	Data, instrument, record, and known-empty coverage writes
DataCatalog	Combined read and write capabilities
Query objects and typed selectors	Explicit data families, identifiers, time bounds, and filters
Catalog and writer factories	Construction of registered implementations from configuration
CatalogWorker	Serialized catalog access, session ownership, and queued writes

Optional capabilities return a distinguishable unsupported-operation error. A common interface does not imply that every backend supports every operation.

Parquet remains the default and only built-in catalog backend in this proposal. External Rust implementations can register factories without requiring runtime consumers to depend on their concrete types. Built-in data, records, and instruments retain typed schemas across implementations.

The shared query types also contain historical-selection surfaces such as CatalogAsOf. Parquet supports reading the latest data and rejects historical selection; it provides no snapshot or time-travel guarantee. The scope and stability of interfaces intended for future backends remain a separate acceptance question before release.

Batches and query sessions

DataBatch represents a collection from one data family. BatchView<T> provides shared storage and slicing, while DataRef provides borrowed access to individual items.

This extends the batching direction from the earlier RFC into catalog queries. It does not make persistence a dependency of the model or in-memory replay foundation.

Parquet query sessions produce timestamp-aligned batches. A requested chunk size is a target: a batch can grow to include rows sharing its boundary timestamp. Instrument and custom-data fallback paths collect results before chunking, so the API does not promise bounded-memory streaming for every query family.

The implementation adapts catalog batches into rows at the existing replay boundary. End-to-end borrowed replay and broader data-engine loading changes remain separate work. Batch boundaries must preserve replay order and logical event boundaries, including atomic OrderBookDeltas events.

This proposal also includes instruments in Data, DataRef, and DataBatch, allowing instrument results to use the shared batch and dispatch interfaces. The earlier RFC explicitly deferred this decision. Adding Data::Instrument requires updates to exhaustive matches, and instrument events entering replay must follow its ordering rules.

Workers and coverage

CatalogWorker serializes access through a bounded command queue. The queue applies backpressure when callers submit work faster than the catalog can process it. Completed sessions close automatically, and flush reports accumulated asynchronous write errors.

Coverage distinguishes an interval that has been queried and contains no data from an interval that has not been covered. Migration must preserve these known-empty intervals alongside stored rows.

Arrow schema v2

The revised schemas use standard Arrow representations for values that previously required Nautilus-specific interpretation.

Value	Proposed representation	Consequence
Prices and sizes	Decimal128(38, 16)	Exact decimals within the supported range
Instant timestamps	Nanosecond timestamps with UTC timezone	Explicit timestamp semantics
Model enums	Dictionary<Int8, Utf8> containing enum names	Readable values instead of numeric codes
JSON fields	UTF-8 with the arrow.json annotation	Explicit JSON semantics
Custom Money	Struct with decimal amount and currency dictionary	Amount and currency remain distinct
Depth sides	Lists of price, size, count, and order ID structs	Variable depth and preservation of order IDs

Undefined prices and quantities use Arrow nulls. Invalid values, unsupported precision, and values outside the representable range must produce errors rather than silent rounding or reinterpretation. Price and quantity precision above 16 is outside the proposed catalog representation. Durations remain integer values.

DeFi precision requirements, including wei and values requiring 18 decimal places, will be revisited separately. A separate DeFi-specific schema is a likely approach; its representation and scope remain TBD.

Display output is a separate convenience representation and can use floating-point prices and sizes. Consumers requiring exact decimal values must use raw output where supported.

Depth and custom data

Variable-depth snapshots

OrderBookDepth replaces the fixed-length model with owned, variable-length sides. The OrderBookDepth10 name remains a compatibility alias, but callers must handle empty sides, unequal side lengths, and the removal of padding by general constructors.

Arrow and Cap’n Proto representations preserve all supplied levels and order IDs. Legacy C and SBE representations retain their fixed-depth constraints; incompatible shapes must fail conversion. Order IDs omitted by older stored formats cannot be recovered during migration.

Custom-data generation

Custom-data generation separates model behavior from persistence:

• nautilus_model::custom_data provides model traits, JSON conversion, and optional Python methods.
• nautilus_serialization::arrow_custom_data provides Arrow schemas, codecs, and optional PyArrow methods.

Model-only types can therefore avoid Arrow dependencies. Types requiring persistence need the corresponding Arrow support and registration. The macro crate changes from nautilus-persistence-macros to nautilus-macros.

Existing adapter and user-defined schemas require explicit migration coverage.

Streaming and analytics

Feather remains the default streaming writer. The opt-in Parquet writer stages data in Feather and promotes sealed files into the catalog.

Flushing staging does not by itself guarantee that data is queryable. Promotion can occur explicitly, at a configured interval, or on close. Close-time promotion defaults to enabled, while deletion of committed Feather source files remains opt-in.

Recorded promotion identities prevent repeating conversions already recorded as complete. These operations do not provide an atomic multi-file catalog transaction. The streaming guide must explain visibility, failure recovery, and retry behavior.

The Python query_catalog API supports PyArrow, pandas, Polars, and DuckDB outputs through Arrow C streams. Results group identities lexically and preserve order within each group. Consumers needing global chronological order must sort explicitly; this query presentation contract does not change replay ordering.

Downstream conversions have their own precision and dtype behavior. Raw decimal columns become Python Decimal objects in pandas unless Arrow-backed dtypes are requested. DuckDB converts timezone-aware nanosecond timestamps to microsecond TIMESTAMPTZ, losing sub-microsecond precision, as described in its timestamp documentation.

Breaking changes

This transition requires both persisted-data migration and updates to affected integrations.

Surface	Compatibility impact
Existing catalogs	Explicit conversion to a separate destination
External Arrow readers	Updated column types, null handling, enums, and depth shapes
Catalog layout	Normalized type directories and custom-data identifier paths
Depth callers	Variable-length owned sides instead of fixed ten-element storage
Rust data variants	Data::BookDepth10 becomes Data::BookDepth; Instrument is added
Catalog integrations	Shared interfaces, typed queries, and changed configuration APIs
Python integrations	Updated configuration signatures and depth wrangler result types
Custom data	Split macros, crate rename, and revised persisted schemas
Display consumers	Nested depth lists and display-specific numeric representations

The Python migration examples must cover catalog and streaming configuration, including writer selection and rotation settings. The depth wrangler returns OrderBookDepth values. Additive APIs such as query_catalog should be documented separately from changes requiring existing callers to adapt.

The migration guide must also cover narrower observable changes: rejection of explicit empty streaming filters, enforcement of the existing multithreaded-runtime requirement for set_runtime, corrected high-precision numeric text, and type-level custom subscriptions receiving identifier-scoped payloads.

Compatibility aliases and retained import paths reduce disruption where available. They do not make the complete transition backward compatible.

Migration support

The PR includes nautilus catalog migrate-parquet and a migration guide. The proposal is to complete and validate this supported path.

The conversion contract is:

• Preserve source files, including when conversion fails.
• Write to a separate, empty destination using create-only writes.
• Reject overlapping source and destination locations.
• Provide a dry run for schema and layout checks.
• Report unsupported formats explicitly.
• Preserve identifiers, exact values, timestamps, ordering, and known-empty coverage.

The command does not provide in-place migration or resumable conversion into a partially populated destination. Feather streams are outside its scope. The guide must provide a verified procedure for existing staged runs rather than assume the catalog migration command handles them.

The migration planner identifies supported schemas by fingerprint. Before release, the compatibility policy must settle whether fingerprints remain the identification mechanism or files also carry explicit version metadata. Runtime readers must reject unsupported legacy schemas before decoding rows and provide an error directing users to the migration command.

The completed guide should name the supported source releases and schema families, storage requirements, custom-type registration requirements where applicable, and application API changes. Its support matrix must distinguish tested adapter-specific conversions from unsupported or unverified cases. For example, the implementation reports portfolio_snapshot directories as unmigrated.

Validation must compare decoded values and identities, not only file or row counts. It must include representative catalogs written by supported released versions in both standard- and high-precision builds.

The guide should explain how to validate the destination before cutover and how to return to the preserved source with a compatible application version. No reverse migration is provided: returning to the old catalog does not carry across data written only to the v2 destination after cutover.

Release requirements

Integration into develop can precede release readiness. A release containing this transition should require:

• A documented schema identification and compatibility policy.
• A complete breaking-change inventory with Rust and Python migration examples.
• Conversion tests using supported released-version catalogs in both precision builds, including adapter custom data.
• Tests that reject unsupported legacy schemas with a clear migration error.
• Exact round-trip tests across supported numeric build configurations.
• Query coverage for empty depth sides, instrument time bounds, and raw/display output.
• Promotion tests covering multiple instruments, every supported record family, and failure recovery.
• Migration tests covering partition identity, unsupported schemas, source preservation, and partial failure.
• Replay parity tests showing that batching preserves ordering and event boundaries.

Known correctness defects are implementation work to resolve before release, not accepted changes to the data contract.

Alternatives and decisions requested

Keeping the existing binary schemas avoids immediate migration but retains Nautilus-specific decoding and build-dependent storage representations.

Supporting old and new schemas in ordinary runtime queries would reduce immediate conversion work, but would also extend compatibility logic into every read path. This proposal favors explicit migration and a single runtime schema contract.

The requested decision is to adopt the shared catalog interfaces and Arrow schema v2, including instruments in the shared data representations, with migration support as part of the release requirement.

Before release, the proposal also needs agreement on:

• Schema identification and legacy-read errors.
• The earliest supported migration source release and covered schema families.
• The stability of interfaces intended for future backends.
• The retention window for compatibility aliases such as OrderBookDepth10.

Feedback is particularly useful on legacy and custom schemas that need support and integrations affected by the API changes. DeFi-specific numeric representation remains separate follow-up work.