Audio-text retrieval eval (AudioCaps/Clotho) and audio front-end unit tests

The gap

The audio path now has a solid zero-shot classification eval (open_clip_train/audio_zero_shot.py, scripts/clap_zeroshot.py, ESC-50 style, Hugging Face datasets, works for both HTSAT CLAP and NaFlexClap), but there is no retrieval eval in the tree: neither "audiocaps" nor "clotho" appears anywhere in the repo. Audio-text retrieval R@k on AudioCaps and Clotho is the standard way CLAP-family checkpoints are compared in the literature, and it is the benchmark that #1137's own test plan had to reach for an external CLIP Benchmark fork to run.

That matters for two of the requirements raised in the #1137 review:

• "should be able to easily run zero-shot during train (or stand alone from main) as we do with imagenet, ideally without a 3rd party dependency like CLIP_benchmark"
• "need pretrained checkpoints that load with results that can be confirmed with zero-shot eval"

Classification zero-shot alone cannot confirm a CLAP checkpoint against published numbers, because the numbers people publish for these models are mostly retrieval. An in-repo retrieval eval closes that loop without CLIP_benchmark, and also gives the release verification pass mentioned in #1191 a way to check audio additions against known results once weights exist.

Proposal

1. Audio-text retrieval eval mirroring the existing structure. A open_clip_train/audio_retrieval.py module shaped like audio_zero_shot.py: Hugging Face dataset in, --audio-retrieval-* args, runs inside zero_shot_eval_all during training and standalone via a scripts/clap_retrieval.py twin of clap_zeroshot.py. Grouped a2t/t2a recall (each clip has multiple reference captions, COCO convention: an audio query scores its best-ranked reference caption), metric names following metrics._add_rank_metrics (audio_to_text_R@{1,5,10}, mean/median rank). Tower-agnostic through the same model(audio=...) / model(text=...) calls the training loop uses, with the eval transform chosen exactly the way the zero-shot path chooses it (NaFlex mel-patchify vs fixed-clip AudioPreprocess), so HTSAT CLAP and NaFlexClap are both covered by one code path.

   Dataset side: public HF mirrors exist for both benchmarks (an AudioCaps mirror with caption-level rows keyed by youtube_id, and a Clotho mirror whose test split matches the canonical 1045-clip evaluation set with 5 captions per row). The loader takes dataset id plus column names rather than hardcoding one mirror, documents provenance caveats (AudioCaps mirrors are smaller than the canonical 975-clip test gallery because of YouTube attrition), and can write the exact clip ids it scored to JSON so results are auditable against training manifests. No new default dependencies; datasets is already in the audio extra and is imported lazily, same as the zero-shot path.

2. Unit tests for the audio front-end. CPU-only, synthetic waveforms, no dataset downloads, matching the "would be good to have unit tests working some some audio coverage as that helps with further changes" comment on #1137. Concretely: mel orientation pins for both extractors (the CLAP extractor is time-major, the Whisper one frequency-major, which is exactly the kind of thing that silently transposes), a regression test for the mel-patch within-patch flatten order that the 2026-06-24 layout fix corrected, and resampling correctness against analytic references with explicit tolerances. The tolerance test is motivated by a real failure mode: in serving work on our own audio embedding models we measured a lossy resampling path degrade real clips to 0.972 cosine similarity against the reference pipeline, which is enough to move retrieval metrics.

What this does not try to do

It does not add weights, does not touch model configs or the training loss path, and does not add dependencies. It is eval plus tests only, so it should be low risk to merge and useful for verifying any future audio checkpoints.

Offer

I have a working implementation of both parts (eval module, standalone script, train-loop wiring, and the test files, all passing CPU-only) and can open a PR if the shape sounds right. One open question where your preference matters: the accessible AudioCaps mirrors are community re-uploads (CC-BY-NC, and smaller than the canonical test set due to YouTube attrition), so if there is a source you would rather treat as canonical, the loader can default to it. Happy to adjust dataset handling, metric naming, or scope to whatever fits the release plans.

Background: we build and evaluate audio-text embedding models at Eximius Labs (fusion-embedding-2, evaluated on AudioCaps/Clotho retrieval; arXiv:2607.18666), so we run into these eval and front-end pitfalls regularly.