Consider an opt-in ultrafast-pycocotools backend for COCO evaluation
Search before asking
- I searched existing issues and pull requests for this backend and found no matching feature request.
Description
Would you consider an opt-in ultrafast-pycocotools evaluator for the existing COCO evaluation path? I maintain the library and am requesting agreement on the approach before submitting a feature PR, as required by the contribution guidelines.
The library is a BSD-2-Clause Rust/Python COCO evaluator, available on PyPI:
python -m pip install ultrafast-pycocotools==0.1.5
Version 0.1.5 provides tested CPython 3.9–3.14 wheels for Linux x86-64, Windows x86-64, and macOS Intel/Apple Silicon. Ultralytics' existing Python 3.8 support should remain on the current backend; this proposal would not raise the core Python requirement or add a mandatory dependency.
Version 0.1.5 fixes source-archive license packaging; the evaluator implementation is unchanged from the measured 0.1.4.
Use case
Reduce COCO evaluation time and peak host memory during validation, particularly for saved predictions and repeated large evaluations. I inspected DetectionValidator.coco_evaluate at revision 88b030173dbbaff2f22925faffd0cb780500c1a9, which already uses faster-coco-eval and also serves segmentation and pose validation.
The proposed change would stay in this shared owner, keep faster-coco-eval as the default, and permit explicit selection of ultrafast. I would avoid a separate validator implementation and a new global CLI option unless maintainers prefer one. Existing metric keys, image selection, cached GT behavior, and fitness computation must remain unchanged. COCO detection/segmentation/pose and LVIS behavior would need focused validation before any PR is marked ready.
Evidence and limits
A saved-prediction benchmark using YOLO26n on all 5,000 COCO val2017 images (596,202 detections) produced:
| Backend | Scoring time | Whole-process peak RSS |
|---|---|---|
| pycocotools 2.0.11 | 37.073 s | 1,599.2 MiB |
| faster-coco-eval 1.8.0 | 7.257 s | 1,651.6 MiB |
| ultrafast 0.1.2 | 1.942 s | 745.9 MiB |
This historical 0.1.2 experiment is a single fresh process per backend, not a repeated estimate for the newly published 0.1.5. It includes GT indexing/result loading/evaluation/accumulation/summary, excludes JSON parsing and inference, and includes parsed inputs in peak RSS. It does not measure the complete current Ultralytics validator. All ultrafast precision, recall, scores, and stats arrays match pycocotools byte for byte. Recorded inputs, versions, timings, and output hashes.
The 0.1.4 paired report separately measures YOLO26n file evaluation at 0.628 s and 223.2 MiB median over three runs, including JSON loading. It compares 0.1.3 with 0.1.4; these different timing scopes should not be mixed to claim a speedup against the table above.
As additional framework evidence, actual RF-DETR one-pass metric replay with 0.1.4 improved from 14.130 s / 3,301.7 MiB to 6.747 s / 2,073.2 MiB versus faster-coco-eval, with exactly equal output metric tensors. That result belongs to RF-DETR and is not a claimed Ultralytics speedup.
Additional
- Library and reproducible benchmarks
- Compatibility and CI coverage
- The library exposes the metric-key aliases, logging callback, and LVIS mode used by the current evaluator. Full integration should still be assessed against real supported task paths and version floors, rather than inferred from API names alone.
- I would first agree on a minimal selection mechanism and validation requirements, then submit a small PR. Default replacement could be considered separately after broader validation.
Are you willing to submit a PR?
- Yes, I would like to submit a PR after approval of the approach.
Source: ultralytics/ultralytics