Consider an opt-in ultrafast COCO metric backend: 2.09x faster with 37% less peak RSS

Search before asking

• I searched existing issues and pull requests for this backend and found no matching request.

Description

Would you consider an opt-in ultrafast-pycocotools backend for RF-DETR's one-pass COCO metric? I maintain the library and would like approval on the integration approach before submitting a feature PR, following the repository's contribution guidance.

The package is available on PyPI under BSD-2-Clause, with tested CPython 3.9–3.14 wheels for Linux x86-64, Windows x86-64, and macOS Intel/Apple Silicon:

python -m pip install ultrafast-pycocotools==0.1.5

I inspected RF-DETR revision 39c2d3a26be81d863abbe84e153445c7813241e5. The training callback already uses faster-coco-eval through OnePassCocoMeanAveragePrecision; aliasing pycocotools alone does not replace this path.

Version 0.1.5 fixes source-archive license packaging; the evaluator implementation is unchanged from the measured 0.1.4.

Use case and measured evidence

Using actual RF-DETR Nano predictions on all 5,000 COCO val2017 images (1,500,000 detections), I replayed identical CPU tensors through the real upstream metric with class_metrics=True and maxDets [1, 10, 500]:

That is 2.09× faster with 37.2% less peak RSS for this metric component. Every aggregate/per-class metric tensor and class-ID tensor matches both the current backend and a pycocotools reference control byte for byte.

Three fresh processes per timed backend, alternating order; shared AMD EPYC 9554, two CPU threads, Python 3.12.3, NumPy 2.4.4, TorchMetrics 1.8.2. Time covers construction, updates, single-process merge, and compute; excludes inference and prebuilt tensor-trace loading. Peak RSS includes framework imports and inputs. These are metric-component measurements, not training-epoch speedups.

• Report, timings, exact input/output hashes, and reproduction commands
• Benchmark graph
• Per-instance adapter
• Integration tests

Proposed PR scope

Keep faster-coco-eval as the default and make ultrafast an explicit optional selection. Apply the existing instance adapter consistently to the validation/test, train, and EMA metric constructors, before any updates. No global import replacement, model changes, or changes to F1/checkpoint selection.

The adapter preserves RF-DETR's AP summary convention at maxDets=500 and converts TorchMetrics boolean masks to the uint8 layout expected by the COCO mask API. It retains the upstream private-contract check and distributed-state lifecycle. The current faster-coco-eval dependency remains because RF-DETR's metric constructor requires it.

Tests cover bbox/segm/combined metrics, maxDets 100/500, crowds, score ties, empty prediction batches, per-class tensors, pickle, reset/reuse, rollback on contract failure, and two-rank CPU/Gloo state merging. Full-dataset timing is bbox only; real segmentation workloads and multi-GPU training validation would still need upstream validation. Legacy CocoEvaluator and keypoint evaluation would remain unchanged.

Would an optional backend in the current metric/callback be welcome, and where would you prefer the selection to be exposed? I would keep the first PR limited to the approved interface and its integration coverage.

Are you willing to submit a PR?

• Yes, I would like to submit a PR after agreement on the approach.