[Bug]: One environment's constraint NaN kills the whole batched scene — no per-env fault isolation or failing-env index
Bug Description
With scene.build(n_envs=N), when any single environment's rigid solver produces invalid constraint forces, scene.step() raises GenesisException: Invalid constraint forces causing 'nan' for the entire scene. Two problems compound:
- No quarantine — at
n_envs=4096, one bad contact configuration destroys 4,095 healthy rollouts mid-step. - No environment index — the exception carries no indication of which environment went non-finite, so the failing rollout cannot be excluded, reset, or even analysed.
We hit this repeatedly in a contact-rich manipulation batch (16-link articulated cable + Franka, 16–64 envs, gs.amdgpu backend): one environment's gripper/floor/cable interaction goes NaN and the full batch dies.
Steps to Reproduce
The failure is contact-dependent and, on this stack, not deterministic — from bit-identical resets we measured contact rollouts diverging up to 256 mm in final cable position, so the repro is statistical rather than a fixed script. The reliable recipe:
- Build a batched scene (16+ envs) with an articulated chain (~16 links) and a Franka,
scene.build(n_envs=N). - Run a force-controlled grasp-and-drag policy against the chain so fingers, chain, and floor stay in sustained contact.
- Within a few hundred steps, some run of the batch raises
GenesisExceptionfromscene.step()for all environments.
Happy to provide our full scene setup and driver if useful (public at Enoch208/Crux, src/crux/simulation/batchscene.py).
Expected Behavior
Per-environment fault isolation — quarantine the offending environment (freeze it, flag it, or reset it) and let the remaining N−1 continue; at minimum, include the failing environment index (or a per-env validity mask) in the exception so callers can salvage and analyse.
Relevant log output
gs.GenesisException: Invalid constraint forces causing 'nan'.(raised by scene.step(); our batch runner catches it, records every in-flight episode as failed-unstable, and salvages already-finished episodes — but the healthy rollouts are unrecoverable)
Environment
- OS: Ubuntu 24.04.4 LTS
- GPU/CPU: AMD Radeon PRO W7900 (gfx1100), 48 GB
- GPU-driver version: amdgpu, ROCm 7.2.1 (
torch 2.13.0+rocm7.2) - CUDA / CUDA-toolkit version: N/A (ROCm)
Release version or Commit ID
1.3.1
Additional Context
Large-batch evaluation is the headline use-case for GPU simulation, and blast-radius-of-one-env makes long unattended sweeps fragile: our 4,096-env throughput runs were only completable because episodes that finished before the explosion could be salvaged. Related in spirit to #2652 (NaN crash in the manipulation example), but this report is about the batching API contract rather than any specific NaN source.
Source: Genesis-Embodied-AI/genesis-world