[Bug]: Headless Newton/Kit video records stale poses after #7642
Regression
After #7642, headless Newton simulation with on-demand Kit viewport recording can produce nonblank videos with stale robot/body poses while physics continues normally. The recorder can appear healthy even though the video does not reflect the simulated state.
@AntoineRichard @matthewtrepte @StafaH — tagging the author and approving reviewers of #7642 for context on the intended capture contract.
Trigger and cause
The reproduced configuration uses Newton/MuJoCo-Warp physics, a headless KitVisualizerCfg, and Isaac Lab's native VideoRecorderCfg(source="visualizer:kit"), without IsaacSim RTX camera sensors or another continuous visualizer keeping sim.is_rendering true.
#7642 makes headless capture-only visualizers on-demand and adds this capture-time refresh:
if not sim.is_rendering:
sim.forward()
frame = viz.render_rgb_array()For PhysX, forward() refreshes Fabric. For Newton, it performs kinematic/reset-state updates but does not call pre_render(). Newton publishes body transforms, cables, and particles to Kit's rendering stage through pre_render(), called by sim.render(). The direct visualizer capture therefore reads stale published poses.
This is a visualization regression, not evidence of incorrect physics. Active IsaacSim RTX camera sensors keep continuous rendering enabled and avoid this trigger. We have not established the same GPU failure for every standalone visualizer or camera renderer.
GPU validation
Validated on an RTX 6000 Ada with a headless articulated scene: 48 environments, the same seed, two training updates, and an eight-frame native training capture.
| Runtime | Observed video |
|---|---|
Previous Isaac Lab pin a8b4da3c29ae528b39d4b3c9444d782ce58d886d |
Correct reset/bent robot poses |
Develop 7a7575df8746080a5c10f14d3727a550aa9da45a, Newton 1.6.0 |
Stale initial/upright poses |
Same develop revision and dependencies, replacing capture-time sim.forward() with sim.render() |
Correct reset poses restored |
The proposed fix was GPU-tested, not just inferred from a mock. The existing unit regression was also corrected to distinguish kinematic updates from renderer-state publication: it fails with the original implementation and passes with the fix; all 34 recorder tests pass. Checking only for nonblack or changing pixels is insufficient because materials/rendering can change while body poses remain stale.
Possible solution and downstream compatibility
At the native recorder's existing capture boundary:
if not sim.is_rendering:
sim.render()
frame = viz.render_rgb_array()This runs backend render synchronization and visualizer updates when a frame is requested, preserving the idle-window optimization. Possible solution: #7864 implements this capture-boundary fix and documents the custom-recorder requirement. It contains the GPU-validated native-recorder change; it does not automatically repair downstream custom recorders.
Fixing the native recorder alone does not fix existing downstream recorders. Custom recorders that directly call render_rgb_array() or render_tiled_rgb_array() and relied on the previous per-step visualizer updates must now explicitly refresh render state before capture (unless their caller already does so). Adding only sim.forward() is insufficient for Newton. This behavior change can therefore introduce additional downstream regressions even after the native recorder is fixed.
Please consider whether that migration requirement is the intended public contract, or whether a shared capture entry point should own synchronization. The narrow fix addresses the validated native path; downstream compatibility should remain part of the resolution.
Source: isaac-sim/IsaacLab