RandomMosaic: boxes ignore output_size, unselected samples gain phantom [0,0,1,1] boxes, resample mode crashes with output_size=None
Summary
Four RandomMosaic defects around output_size, unselected samples and boxes. #4438 / #4468 (closed) fixed the axis swaps; these remain.
import torch, kornia.augmentation as K
img = torch.rand(4, 1, 6, 8); boxes = torch.tensor([[[1., 1., 4., 4.]]] * 4)1. Boxes ignore output_size. output_size=(4, 10) and output_size=None return identical boxes; e.g. [0.0, 4.4613, 1.0, 6.0] has y2 = 6 in an image of height 4.
for osz in (None, (4, 10)):
torch.manual_seed(2); y, b = K.RandomMosaic(output_size=osz, p=.5, data_keys=["input", "bbox_xyxy"])(img, boxes)
print(tuple(y.shape), b[1])2. Unselected samples gain phantom boxes. With batch_prob = [0, 1, 0, 1] (seed 2 above), row 0 comes back as [[1,1,4,4],[0,0,1,1],[0,0,1,1],[0,0,1,1]]: three [0,0,1,1] boxes that were never in the input and that min_bbox_size does not remove. The box count also depends on the gate: (B, 1, 4) at p=0 vs (B, 4, 4) otherwise.
3. An unselected sample is not returned unchanged when output_size is set. K.RandomMosaic(output_size=(4, 10), p=0.)(torch.rand(3, 1, 6, 8)) has shape (3, 1, 4, 10): the input is zero-padded / cropped (values dropped) to output_size. Possibly unavoidable for a batched tensor, but then it should be a documented, tested contract, and the two cropping_modes should agree on it (slice interpolates an input-sized crop to output_size, resample takes an output_size window without rescaling).
4. cropping_mode="resample" with the default output_size=None crashes.
K.RandomMosaic(p=1., cropping_mode="resample")(img)
# TypeError: 'NoneType' object is not subscriptable (imgwarp.py, dsize[0])Also, for the record: mosaic_grid is ordered (tiles along width, tiles along height), while RandomJigsaw.grid is (rows, columns).
Measured on main @ c643312a3 merged with #4634 (docs-only), torch 2.14.0, CPU, macOS arm64. Found while reviewing #4634 (batch 6d of #4407).
Posted on behalf of @ducha-aiki by Claude (Fable 5.1).
Source: kornia/kornia