[SAM 3.1 Multiplex] Explicit forward tracking bound can enter an empty grounding chunk

Summary

In the SAM 3.1 Multiplex video path, an explicit forward max_frame_num_to_track is interpreted with two different boundary conventions:

• _get_processing_order treats it as an inclusive offset and schedules N + 1 frames.
• forward_video_grounding_batched_multigpu uses it to form an exclusive right bound and prepares only N grounding inputs.

When propagation reaches the extra final frame, the grounding chunk is empty (with grounding batch size 1), and _batch_find_inputs raises IndexError: list index out of range.

I reproduced this against main at 660a5e9e1b8b4c02c0ad97229b88a09a6e4ff5b7 with the SAM 3.1 Multiplex checkpoint.

Reproduction

1. Initialize a SAM 3.1 Multiplex video session containing at least 16 frames.
2. Set predictor.model.batched_grounding_batch_size = 1.
3. Add a text prompt on frame 0.
4. Run forward propagation with an explicit bound smaller than the remaining video:

request = {
    type: propagate_in_video,
    session_id: session_id,
    propagation_direction: forward,
    start_frame_index: 0,
    max_frame_num_to_track: 15,
}

for response in predictor.handle_stream_request(request):
    pass

The progress reaches 15/16 and then fails:

Running full VG propagation (reverse=False).
propagate_in_video: 94%|...| 15/16

File sam3_multiplex_tracking.py, line 351, in propagate_in_video
  out = self._run_single_frame_inference(...)
File sam3_multiplex_detector.py, line 750, in forward_video_grounding_batched_multigpu
  chunk_outputs = self._process_grounding_chunk_batched(...)
File sam3_multiplex_detector.py, line 809, in _process_grounding_chunk_batched
  batched_find_input = self._batch_find_inputs(...)
File sam3_multiplex_detector.py, line 140, in _batch_find_inputs
  device = chunk_find_inputs[0].img_ids.device
IndexError: list index out of range

Cause

For start_frame_idx=0 and max_frame_num_to_track=15, the outer tracking loop schedules frames 0 through 15 (16 frames):

https://github.com/facebookresearch/sam3/blob/660a5e9e1b8b4c02c0ad97229b88a09a6e4ff5b7/sam3/model/sam3_multiplex_tracking.py#L295-L297

end_frame_idx = start_frame_idx + max_frame_num_to_track
processing_order = range(start_frame_idx, end_frame_idx + 1)

However, grounding defines valid_frame_end as an exclusive bound of 15:

https://github.com/facebookresearch/sam3/blob/660a5e9e1b8b4c02c0ad97229b88a09a6e4ff5b7/sam3/model/sam3_multiplex_detector.py#L727-L742

valid_frame_end = propagate_in_video_start_frame_idx + max_frame_num_to_track
chunk_end = min(chunk_start + batch_size, valid_frame_end)

On outer-loop frame 15 with batch_size=1, both chunk_start and chunk_end are 15. This creates an empty chunk_find_inputs, which is indexed unconditionally here:

https://github.com/facebookresearch/sam3/blob/660a5e9e1b8b4c02c0ad97229b88a09a6e4ff5b7/sam3/model/sam3_multiplex_detector.py#L133-L140

The mismatch exists regardless of batch size, although a larger batch can make the eventual failure appear at a different indexing point.

Expected behavior

The outer propagation order and grounding range should use one consistent convention. Based on the parameter name and the default assignment max_frame_num_to_track = num_frames, I would expect a value of 15 to process exactly 15 frames, including the start frame (frames 0 through 14).

One possible forward-path correction is:

end_frame_idx = start_frame_idx + max_frame_num_to_track - 1
end_frame_idx = min(end_frame_idx, num_frames - 1)
processing_order = range(start_frame_idx, end_frame_idx + 1)

The reverse path should be reviewed at the same time so both directions have documented, symmetric semantics. Merely ignoring the empty chunk would hide the boundary mismatch and can also leave results buffered by hotstart_delay instead of returning a complete propagation result.

Current workaround

We wrap _get_processing_order so that only the outer loop receives max_frame_num_to_track - 1, while the original full value remains in feature_cache[tracking_bounds] for grounding. This prevents the extra iteration without shortening the grounding input range.