[Persistence] [v2] Custom Feather conversion silently writes no Parquet data

Bug Report

Confirmation

• I've re-read the relevant sections of the documentation.
• I've searched existing issues and discussions to avoid duplicates.
• I've reviewed or skimmed the source code and originating pull requests to confirm the behavior is not by design.
• I've tested this issue using a recent pre-release or development wheel (2.0.0rcN, dev develop, or a nightly) and can still reproduce it.

I reproduced this from source against current develop commit 45903fc8b925adae6323035fb0b4fb5b49b4f89b. I have not separately repeated the reproduction against a packaged development wheel.

I searched issues and pull requests, including closed and merged results, for custom data parquet, convert_stream_to_data, Feather custom data, to_snake_case custom, and custom parquet conversion. I found related work, including #3542, #3590, #1796, #3954, and #4297, but no existing report for this specific marker and stream-layout defect.

This appears to be a bug rather than an enhancement request. Merged PR #3542 introduced the custom-data persistence architecture and explicitly included conversion of custom data written through the Feather writer into Parquet. docs/concepts/custom_data.md likewise states that the custom-data conversion branch writes Feather stream data to Parquet. The implementation also contains a data_cls.starts_with("custom/") conversion branch. That branch is currently unreachable for Feather files written by FeatherWriter.

Expected behavior

When registered custom data is written through FeatherWriter, calling:

catalog.convert_stream_to_data(
    instance_id,
    "custom/RustTestHashMapCustomData",
    Some("live"),
    None,
    false,
)?;

should discover the generated Feather files, convert their Arrow batches into Parquet, and preserve the custom type and identifier layout.

A subsequent query for RustTestHashMapCustomData should return every record written to the Feather stream. For example, writing three records for each of two identifiers should result in six queryable Parquet records.

The expected flow is:

1. FeatherWriter writes the stream under live/{instance_id}/data/custom/{TypeName}/{identifier}/....
2. convert_stream_to_data recognizes the logical name custom/{TypeName}.
3. Stream discovery maps that logical name to the physical data/custom/{TypeName} directory.
4. The Arrow batches are written under the catalog’s normal data/custom/{TypeName}/{identifier}/... Parquet layout.
5. query_custom_data or query_custom_data_dynamic recovers the original records.

Actual behavior

convert_stream_to_data returns Ok(()), but it discovers no Feather files, writes no Parquet data, and a subsequent custom-data query returns zero records.

There are two coordinated causes.

First, convert_stream_to_data normalizes every data_cls using to_snake_case:

let data_name = to_snake_case(data_cls);

For a custom type, this changes:

custom/RustTestHashMapCustomData

into:

custom_rust_test_hash_map_custom_data

This destroys the custom/ marker used by is_supported_stream_data_type and the custom conversion branch.

Second, even if the marker is preserved, list_feather_files searches beneath:

{root}/live/{instance_id}/custom/{TypeName}/...

while FeatherWriter::get_writer_path_custom physically writes beneath:

{root}/live/{instance_id}/data/custom/{TypeName}/...

Consequently, no data_name currently produced by convert_stream_to_data reaches both the intended custom-data branch and the actual Feather directory.

The failure is silent because the method deliberately treats an empty file list as a successful no-op:

if feather_files.is_empty() {
    return Ok(());
}

That behavior is reasonable when a supported stream genuinely has no files, but here it masks the naming and layout mismatch. The caller receives success while all custom records remain unconverted.

Steps to reproduce

1. Build current develop at 45903fc8b925adae6323035fb0b4fb5b49b4f89b.

2. Register one of the existing Rust test custom-data types, such as RustTestHashMapCustomData.

3. Create a temporary ParquetDataCatalog.

4. Construct a FeatherWriter rooted at:

   复制

   {catalog_root}/live/2026-07-24

5. Write six CustomData records through FeatherWriter: three using identifier ident_a and three using ident_b.

6. Close the writer so the Arrow IPC streams are complete.

7. Confirm that the writer created files beneath:

   复制

   live/2026-07-24/data/custom/RustTestHashMapCustomData/ident_a/
   live/2026-07-24/data/custom/RustTestHashMapCustomData/ident_b/

8. Run:

   rust复制

   catalog.convert_stream_to_data(
       "2026-07-24",
       "custom/RustTestHashMapCustomData",
       Some("live"),
       None,
       false,
   )?;

9. Query the catalog:

   rust复制

   let queried = catalog.query_custom_data_dynamic(
       "RustTestHashMapCustomData",
       None,
       None,
       None,
       None,
       None,
       false,
   )?;
   
   assert_eq!(queried.len(), 6);

10. Observe that conversion returns success but queried.len() is 0.

I also repeated the focused check after preserving the custom/ marker and mapping the logical custom name to the writer’s physical data/custom/... prefix. The same test then recovered all six records.

Code snippets or logs

The essential reproduction is:

ensure_test_custom_data_registered();

let instance_id = "2026-07-24";
let subdirectory = "live";

// Write three RustTestHashMapCustomData records for "ident_a"
// and three for "ident_b" through FeatherWriter. The resulting
// physical layout is:
// live/2026-07-24/data/custom/RustTestHashMapCustomData/{identifier}/...

catalog
    .convert_stream_to_data(
        instance_id,
        "custom/RustTestHashMapCustomData",
        Some(subdirectory),
        None,
        false,
    )
    .unwrap();

let queried = catalog
    .query_custom_data_dynamic(
        "RustTestHashMapCustomData",
        None,
        None,
        None,
        None,
        None,
        false,
    )
    .unwrap();

assert_eq!(queried.len(), 6, "expected all 6 written records back");

Against pristine develop, the focused test fails as follows:

FAIL nautilus-persistence::test_catalog test_convert_stream_to_data_custom_data_daily_roundtrip

thread 'test_convert_stream_to_data_custom_data_daily_roundtrip' panicked at
crates/persistence/tests/test_catalog.rs:
assertion `left == right` failed: expected all 6 written records back
  left: 0
 right: 6

Summary [0.041s] 1 test run: 0 passed, 1 failed, 132 skipped

The method itself returned Ok(()); the assertion failed only when the catalog was queried afterward. This is why the defect can go unnoticed unless the complete Feather-to-Parquet round trip is asserted.

A narrowly scoped correction is to:

1. Preserve custom/{TypeName} verbatim instead of applying to_snake_case to custom names.
2. Keep that value as the logical identity used by custom-data dispatch.
3. Search for custom Feather files beneath the physical data/custom/{TypeName} prefix.
4. Leave built-in names such as quotes, trades, and bars unchanged.

With those two discovery corrections, the focused round-trip test reports:

Summary [0.056s] 1 test run: 1 passed, 132 skipped

Specifications

• OS platform: macOS 26.5.2, build 25F84, Apple Silicon
• Python version: 3.9.6 system Python; reproduction executed as a Rust integration test
• Rust version: rustc 1.97.1 (8bab26f4f 2026-07-14)
• nautilus_trader version: develop commit 45903fc8b925adae6323035fb0b4fb5b49b4f89b
• Installed from: built from source
• Adapter/venue: Not applicable; persistence/catalog layer