RF-DETR Seg + SAHI: still missing cells in dense tiles, and inference is too slow (1900–2200px) — tuning & speedup advice?

Summary

I'm using RF-DETR instance segmentation on relatively large images (e.g. 1900×1400, 2000×2000, 2200×2200) with very dense cells(500~1500 per image).

• RF-DETR inference is fast, but due to the segmentation query/prediction cap it misses many instances in dense scenes.
• SAHI sliced inference improves recall, but:
  1. inference becomes too slow (many tile forward passes + merge/postprocess), and
  2. even with SAHI I still miss some cells in very dense regions (I suspect some tiles still contain too many instances and hit the per-tile cap / filtering, or merge/postprocess suppresses predictions).

My goal is to keep RF-DETR and get SAHI-like recall with much higher throughput, and also understand how to choose SAHI slicing parameters so dense tiles don't miss objects.

---

Input images

• Typical image sizes:
  • 1900×1400
  • 2000×2000
  • 2200×2200
• Dense instances per image: <fill: e.g. 500 ~ 1500+>
• Objects: small cells (crowded / overlapping)

---

Environment

• CPU: Ryzen 9700 (target deployment also includes i7 11th-gen)
• RAM: 64GB (target deployment also includes 16GB)
• GPU: 5090 32GB (target deployment "CPU-only", or low GPU)
• rfdetr model: Nano (recommendations welcome: M/L/XL/2XL)
• Inference runtime: ONNX Runtime in C# (deployment),

---

Current pipeline

1. Resize to model resolution
2. Run SAHI sliced inference
   • slice size: 384×384
   • overlap ratio: 0.1 ~ 0.3
3. Merge tile predictions back to full image

---

Timing (current bottleneck)

• SAHI tiling: ~ 1 hour 22 min per 1000 images
  • (~4.92 sec/image average)

I’d appreciate advice on what to measure to pinpoint the bottleneck:

• model forward vs preprocessing vs merge/postprocess vs Python overhead, etc.

---

Questions

1. SAHI slicing parameters for dense scenes (to avoid misses)

   • How should I choose slice size / overlap so that dense tiles do not still miss cells?
   • Is it expected that if a single tile contains more instances than the seg cap, RF-DETR will still miss some even under SAHI?
   • Any rule-of-thumb for cell-like crowded small objects?

2. Best SAHI merge/postprocess settings for crowded small objects

   • For dense scenes, what postprocess settings are recommended (NMS vs NMM/GREEDYNMM, IOU vs IOS, thresholds) to avoid suppressing true positives?

3. Throughput optimization

   • What are the biggest wins to speed up RF-DETR+SAHI inference?
     • batching tiles on GPU / batching in general
     • reducing merge/postprocess cost
     • choosing slice size to reduce tile count while keeping recall
     • model-side optimizations (fp16, compile, etc.)

4. References / prior art

   • Are there examples/issues/repos where people successfully used RF-DETR + SAHI for dense instance segmentation (or similar crowded small-object setups)?
   • If yes, could you point me to configs or recommended practices?

---

Goal

A practical recipe for:

• SAHI slicing settings that maximize recall for dense tiles (without misses),
• and a recommended optimization path to improve throughput significantly.