[LTX-2.5] DiffVAE AUTO tiling produces gray tails at the 2^32 stage-5 element boundary

Summary

LTX-2.5's diffusion-based video VAE decoder can silently produce neutral-gray frame tails or rectangular gray tiles when AUTO tiling selects a chunked stage-5 tile whose channels-last activation reaches the 2**32 flattened-element boundary.

This is not generated by stage 1 or stage 2. The same finite stage-2 latent decodes cleanly with a smaller explicit tile and reproduces the gray tail with the original AUTO tile.

I reproduced this on an A100 80 GB with the official LTX-2.5 split checkpoints and ltx_pipelines.ti2vid_two_stages_hq.

Environment

• Repository main: fd4ded7f2d88d3da713abcdd4ad41ecc4a9314ca
• GPU: NVIDIA A100-SXM4-80GB
• Driver: 535.54.03
• Python: 3.12.13
• PyTorch: 2.9.1+cu128
• Triton: 3.5.1
• CUDA runtime reported by PyTorch: 12.8
• Output: 1920x1088, 24 fps
• Official LTX-2.5 transformer, text encoder, video VAE, audio VAE, distilled LoRA, and spatial upsampler

This appears distinct from #37: that report was specific to RTX 5090/ComfyUI, whereas this reproduces on A100 and aligns exactly with a deterministic stage-5 flattened-index boundary.

Reproduction

Run the shipped two-stage HQ pipeline with any image/prompt at 1920x1088 and vary --num-frames:

python -m ltx_pipelines.ti2vid_two_stages_hq \
  --transformer-path /path/to/ltx-2.5-22b-dev-transformer-bf16.safetensors \
  --text-encoder-path /path/to/gemma4-12b-with-proj-ltx-2.5-bf16.safetensors \
  --video-vae-path /path/to/ltx-2.5-video-vae-bf16.safetensors \
  --audio-vae-path /path/to/ltx-2.5-audio-vae-bf16.safetensors \
  --distilled-lora /path/to/ltx-2.5-22b-distilled-lora-450-bf16.safetensors \
  --spatial-upsampler-path /path/to/ltx-2.5-latent-spatial-upscaler-x2-bf16-1.0.safetensors \
  --prompt "..." \
  --image input.png 0 1.0 0 \
  --output-path out.mp4 \
  --num-frames 241 --frame-rate 24 \
  --height 1088 --width 1920 \
  --seed 4242 --max-batch-size 1 \
  --distilled-lora-strength-stage-1 0.25 \
  --distilled-lora-strength-stage-2 1.0

Observed duration scan before the fix:

AUTO is resolved from free memory at decode time, so duration alone is not the invariant: different allocator state can select a different tile.

Boundary evidence / root cause

The chunked stage-5 activation is channels-last [T, H, W, C] with C=256 and spatial patch size 4.

For a full-width 1920x1088 tile, stage-5 spatial dimensions are 480x272. Frame 128 starts at:

128 * 272 * 480 * 256 = 4,278,190,080
2**32                         = 4,294,967,296

The boundary lands 49.8% into frame 128, matching the observed approximately 48.6% gray area. Later frames are fully gray.

For a width-1056 tile, stage-5 width is 264. Frame 233 starts at:

233 * 272 * 264 * 256 = 4,283,203,584

The same boundary lands 36.0% into the tile, matching the observed approximately 36.0% gray left tile. Subsequent left tiles are gray.

The gray value is also diagnostic: the unwritten pixel accumulator remains raw zero; mapping decoder output from [-1, 1] to [0, 1] turns zero into 0.5, encoded/decoded as approximately RGB [125, 123, 121] in the tested BT.709 H.264 path.

Stage isolation

For a 241-frame run I saved and validated:

• stage-1 latent: finite, shape [1, 128, 31, 17, 30]
• spatial-upsampled latent: finite, shape [1, 128, 31, 34, 60]
• stage-2 latent: finite, shape [1, 128, 31, 34, 60]

All three decode cleanly for all 241 frames with a safe 112-frame temporal tile. The same stage-2 latent decoded with the original AUTO configuration reproduces gray from frame 233. This isolates the corruption to final DiffVAE decoding rather than either diffusion stage, RNG, or model weights.

Suggested minimal fix

In recommended_decode_tiling_config, reject chunked stage-5 candidates whose effective activation reaches 2**32 flattened elements:

CHUNKED_STAGE5_MAX_ELEMENTS = 1 << 32

effective_t = min(tile_frames, num_frames)
effective_h = min(tile_height, height)
effective_w = min(tile_width, width)

elements = (
    stage5_tokens_for_pixel_tile(
        effective_t,
        effective_h,
        effective_w,
        patch_size=patch_size,
    )
    * stage5_channels
)

if chunked_stage5 and elements >= CHUNKED_STAGE5_MAX_ELEMENTS:
    continue

Capping each candidate by the real content extent matters because a size candidate may be rounded beyond a shorter clip or axis.

I also recommend logging the resolved AUTO tiling configuration. Without it, a run can change behavior with GPU free memory / allocator history and the exact runtime tile cannot be reconstructed from CLI arguments.

Validation of the proposed guard

• 145-frame guarded maximum near-gray fraction: 0.0245%
• 241-frame guarded maximum near-gray fraction: 0.0306%
• Four independent 241-frame / 10-second scenes generated cleanly with the guarded AUTO configuration frames=248, height=1088, width=768
• All outputs passed complete ffmpeg decode

Merge suggestion

I suggest a focused PR containing only:

1. the chunked stage-5 2**32 shape guard in AUTO tiling;
2. unit tests immediately below, immediately above, and with candidates rounded beyond actual content extent;
3. logging of the resolved AUTO tile.

This is low-risk: it only removes unsafe candidate shapes; it does not change model weights, sampling, explicit user-provided tiling, or non-chunked DiffVAE modes.

During investigation I also found a separate long-clip issue in the fallback Triton NA launch grid: assigning T*H to CUDA grid-Y can exceed 65,535. Swapping T*H to grid-X fixes that launch-limit failure, but I recommend reviewing/merging it as a separate PR because it is not the direct cause of the silent gray tail described here.

I have a tested local implementation and regression tests for both changes and can prepare focused PR(s) if maintainers agree with this direction.

https://github.com/user-attachments/assets/3d0c74aa-13e1-4a35-9430-9f366b26ef31