Benchmark OpenVINO batch inference modes on Intel and AMD CPUs

Current status — 2026-09-05

• Default-path mitigation shipped in #25921. Current main 5eda95e83 explicitly selects synchronous full-batch LATENCY in the OpenVINO backend. The retained async implementation is deliberately disabled at its owner and linked to this tracker.
• Remaining work is the native upstream reproduction/artifact handoff and evidence required before reconsidering async execution. The Intel/AMD 240-run matrix below is historical evidence from this issue, not a fresh run in this audit. No current-default hard hang was reproduced here.

Recommend no change to the synchronous default and no deletion/re-enabling of the intentionally paused async implementation. Successful batch-8 async runs sometimes gained throughput, but that does not compensate for three unbounded waits/180-second watchdog timeouts. Re-enablement needs a standalone native reproducer, model/runtime versions and stacks, an upstream resolution, and matched reliability, correctness, and latency measurements on the affected CPUs. This remains a follow-up reliability tracker, not evidence that today's default is critically broken.

Original report and historical context

Summary

The async OpenVINO batch path can hard-hang on both Intel and AMD CPUs. Across 240 isolated runs, every one of the 120 synchronous runs completed, while the async high-throughput modes produced three 180-second timeouts. This independently reproduces the failure class behind scheduled run 32705291304.

PR #25921 forces LATENCY so each complete batch runs synchronously, while retaining the throughput and AsyncInferQueue implementations for future re-evaluation. The results support that as the robust default.

The other three cross-controls (LATENCY + async, THROUGHPUT + sync, and CUMULATIVE_THROUGHPUT + sync) all completed 40/40. They isolate the useful pairing: LATENCY performs best with one full-batch synchronous request; the throughput hints require concurrent requests to use their four streams, which retains the native queue surface that hung.

Performance

Values are medians of five fresh-process repetitions. Each repetition performed 3 warmups and 20 measured inferences. Timing is raw OpenVINO inference, normalized per image; timeout cells compute speed only from completed repetitions.

Intel: INTEL(R) XEON(R) PLATINUM 8581C CPU @ 2.30GHz

AMD: AMD EPYC 9B45

Batch-8 throughput bar charts

L/S = proposed LATENCY + sync, T/A = current THROUGHPUT + async, C/A = CUMULATIVE_THROUGHPUT + async. The T/A FP16 bars each exclude one timed-out repetition; Intel C/A INT8 excludes one timed-out repetition.

xychart-beta
    title "FP16 batch 8 throughput (images/s)"
    x-axis ["Intel L/S", "Intel T/A", "Intel C/A", "AMD L/S", "AMD T/A", "AMD C/A"]
    y-axis "images/s" 0 --> 110
    bar [73.58, 74.15, 74.02, 96.24, 108.70, 108.64]

xychart-beta
    title "INT8 batch 8 throughput (images/s)"
    x-axis ["Intel L/S", "Intel T/A", "Intel C/A", "AMD L/S", "AMD T/A", "AMD C/A"]
    y-axis "images/s" 0 --> 130
    bar [50.39, 61.30, 61.44, 98.44, 126.05, 125.90]

What the CPU information changes

• Intel Emerald Rapids: the Xeon exposes amx_int8, amx_bf16, AVX-512, and VNNI. Ultralytics already detects amx_int8 and reshapes dynamic INT8 OpenVINO models to static before compiling, so the pre-PR backend already chooses synchronous LATENCY for this Intel INT8 case. The raw mode matrix above deliberately bypasses that safeguard to compare runtime strategies; the PR does not impose an Intel INT8 regression on the guarded backend path.
• AMD Turin: the EPYC exposes AVX-512 and VNNI but not AMX. Dynamic FP16 and INT8 batches therefore enter the current async path. At batch 2, the proposed path is 10-45% faster on AMD. At batch 8, successful async runs are 12-22% faster, but the path also independently hard-hung.
• Cumulative throughput is not a third CPU scheduling strategy. OpenVINO documents it as an AUTO mode that adds throughput across multiple devices. Both VMs exposed only CPU; the compiled model reported CPU as its sole execution device, and cumulative results closely tracked ordinary throughput while still producing a timeout. It may be useful as an explicit future option on CPU+GPU/NPU systems, but it is not a safer CPU-only default.

OpenVINO's current guidance also pairs synchronous inference with LATENCY, asynchronous concurrent requests with THROUGHPUT, and reserves CUMULATIVE_THROUGHPUT for using multiple devices: performance hints, automatic device selection.

Timeout evidence

The original CI flake hung in dynamic INT8 batch-2 export-matrix cases on AMD, twice at different model tasks. The independent benchmark hitting the same failure class in FP16 and on Intel shows this is not specific to one quantized model or vendor.

Methodology

• Temporary standard GCE VMs: Intel c4-standard-4 (4 vCPU, 15 GB, Emerald Rapids) and AMD c4d-standard-4 (4 vCPU, 15 GB, Turin).
• Identical software: Ubuntu 24.04, Python 3.13.15, OpenVINO 2026.3.0, PyTorch 2.13.0 CPU, Ultralytics PR branch at 069ca2f86.
• Model: dynamic YOLO26n OpenVINO FP16 and INT8, 640 px, exported at batch 8 with the same COCO8 calibration data.
• Matrix per CPU: 2 precisions x 2 batches x 6 hint/API pairings x 5 randomized fresh-process repetitions = 120 runs.
• Each child had a 180-second watchdog. All VMs were removed after results were copied and validated.

Recommendation

Use the PR's full-batch synchronous LATENCY path as the default. It completed every run, wins batch-2 throughput, is essentially tied for Intel FP16 batch 8, and trades at most 22% successful-run throughput at batch 8 to eliminate the observed unbounded native queue wait. Keep multi-stream async/cumulative execution out of the automatic backend path unless OpenVINO fixes the hang and we can prove it with a substantially larger stress matrix.