#9795·carla

Port NVIDIA Cosmos Transfer support to ue5-dev

Author: youtalkCreated Jun 25, 2026Updated Sep 5, 2026

Current state

NVIDIA Cosmos Transfer integration currently exists only on ue4-dev. It was added there in #9122 and extended in #9297, but it was never brought forward to the UE5 line. As a result, ue5-dev users cannot generate Cosmos Transfer control inputs (AOV frames + HD-map control videos) or drive a Cosmos Transfer server to produce hyper-realistic style-transfer variations of CARLA synthetic data. The ue4-dev implementation cannot be cherry-picked directly: it relies on the UE4 lane/boundary extraction path and a UE4-era tagged-component / stencil flow that no longer exist on ue5-dev.

Goal

Bring full Cosmos Transfer support to ue5-dev with the same end-user workflow as ue4-dev — replay a log in CARLA, capture AOV frames, export the HD-map geometry to Cosmos-compatible JSON, render the HD-map control video, and call a Cosmos Transfer server to restyle the run — while keeping the exported JSON byte-compatible with the existing Cosmos Transfer / RDS-HQ tooling so downstream consumers and the cosmos-drive-dreams utilities work unchanged.

Approach

  • Re-implement lane/boundary extraction on carla::road::Map instead of the UE4-specific path, so the HD-map geometry (lane lines, road boundaries, crosswalks, wait lines, road markings, traffic signs) is derived from the OpenDRIVE road model available on ue5-dev.
  • Port the Stencil subsystem to LibCarla (Stencil, RoadInfoStencil, and the Map / MapBuilder / object-parser support) so stencil/road-info data is available client-side without the removed UE4 tagged-component machinery.
  • Keep the exported JSON byte-compatible with the existing Cosmos Transfer format via dedicated export_cosmos_* exporters and RPCs, validated against the reference artifacts.
  • Ship the work in two phases. Phase 1 delivers the geometry/exporter/pipeline code plus docs and an on-PC capture (AOV frames, HD-map control video, optional server-side restyle), gated on visual verification. Phase 2 covers the remaining polish and the control-sensor work that builds on the Phase 1 foundation.

Implementation plan: four stacked PRs (against ue5-dev)

The implementation will be submitted as four stacked PRs, bottom-up, so each layer can be reviewed independently:

  1. Stencil subsystem → LibCarla: port Stencil, RoadInfoStencil, and the Map / MapBuilder / object-parser support so stencil/road-info data is available client-side without the removed UE4 tagged-component machinery.
  2. export_cosmos_* exporters + RPCs: crosswalks, road boundaries, lane lines, traffic signs, wait lines, and road markings, with the lane/boundary geometry re-derived from carla::road::Map and the exported JSON kept byte-compatible with the existing Cosmos Transfer / RDS-HQ format.
  3. Cosmos Transfer Python pipeline + docs: the branch-independent client/server pipeline (carla_cosmos_gen.py, cosmos_client.py, conda environments, Dockerfiles, example_data), the cosmos-drive-dreams submodule, the ported docs (Docs/nvidia_cosmos_transfer.md + images, mkdocs nav), and the sensor.camera.cosmos_visualization smoke test.
  4. Phase 2 follow-up: the in-engine control sensor and remaining polish that builds on the first three PRs.