Tutorial proposal: compare and reproduce segmentation augmentation samples

Would you be open to a small standalone tutorial showing how to compare two segmentation augmentation policies in ClearML and reproduce the exact image/mask transform behind a reported debug sample?

Proposal Summary

The tutorial would combine three existing ClearML surfaces in one reproducibility workflow:

• Task.connect() selects a bounded light or strong policy and a base seed;
• Logger.report_image() reports the same fixed sample IDs under the same title, series, and iteration in both Tasks; and
• Task.upload_artifact() stores the resolved policy and the recorded realization for each reported sample.

AlbumentationsX would apply each spatial decision to the RGB image and binary segmentation mask together. The example would use small generated fixtures, so it needs no dataset download or model training.

Motivation

When two runs show different augmented samples, the image alone does not identify the complete policy or the sampled parameters that produced it. A configuration alone also does not preserve the random realization visible in the debug sample.

Open issue #1150 describes the need to inspect image debug samples across compared runs. ClearML's current examples document image reporting, configuration tracking, and artifacts separately. The proposed tutorial would show how those surfaces work together for one concrete computer-vision debugging task.

Proposed workflow

1. Generate a few deterministic RGB scenes and binary masks with stable sample IDs.
2. Connect policy_name and base_seed to the ClearML Task.
3. Build a small A.ReplayCompose policy and call it with a stable per-sample invocation_seed.
4. Check that the transformed image and mask have matching dimensions and that the mask still contains only the declared labels.
5. Render an original/transformed image-and-mask contact sheet and report it with stable ClearML keys.
6. Upload policy.to_dict() and a replay manifest as Task artifacts.
7. Replay every recorded realization locally and assert that the reproduced image and mask equal the reported result before the Task completes.

The central calls would remain visible:

task = Task.init(
    project_name="examples",
    task_name="augmentation policy comparison",
)
params = task.connect(
    {"policy_name": "light", "base_seed": 137},
    name="Augmentation",
)

result = policy(
    image=image,
    mask=mask,
    invocation_seed=sample_seed,
)

task.get_logger().report_image(
    title="augmentation-preview",
    series=sample_id,
    iteration=0,
    image=contact_sheet,
)

task.upload_artifact("augmentation_policy", policy.to_dict())
task.upload_artifact("sample_replays", replay_manifest)

The tutorial would intentionally use transforms whose recorded parameters remain JSON-transportable, and it would verify the artifact round trip with A.ReplayCompose.replay(...).

The user can clone the Task, change policy_name, and compare the same sample IDs across both runs. If one result looks suspicious, the policy and replay artifact from that Task reproduce the exact image/mask pair without depending on process order or global RNG state.

Related Discussion

• #1150 requests full-size side-by-side comparison of image debug samples across runs.
• #999 requests direct navigation to the same iteration in compared runs.

Those UI enhancements remain separate. This tutorial uses stable sample keys and one fixed iteration to demonstrate policy provenance and exact replay with the current reporting and artifact APIs.

Scope and placement

This is a tutorial/example proposal. It requires no ClearML API change and would not add AlbumentationsX to ClearML's runtime dependencies.

Would maintainers prefer this as a focused example under examples/reporting, with a matching guide page, or as a standalone documentation tutorial? If the workflow fits the project, I can prepare the tutorial after your guidance on placement.

Optional dependency boundary

The tutorial would install AlbumentationsX only in its own example environment. The current public package is AGPL-3.0-only and requires Python 3.10 or newer. Users install the PyTorch build appropriate for CPU, CUDA, or MPS before AlbumentationsX; PyTorch is intentionally not selected through the AlbumentationsX package metadata.