[Bug]: HETERO:GPU.0,GPU.1 + PIPELINE_PARALLEL silently places whole model on GPU.0 when it fits one card (2x Arc Pro B70, Qwen3.8-27B)

OpenVINO Version

2026.4.0 (OVMS 2026.4.0.869b2186 · OpenVINO backend 2026.4.0-22959-99c81491cc3 · GenAI 2026.4.0.0-3407-7ea2546852a)

Operating System

Windows 11 Pro (build 26200)

Device used for inference

HETERO (HETERO:GPU.0,GPU.1)

Framework

None

Model used

OpenVINO/Qwen3.8-27B-int8-ov

Issue description

With two identical discrete GPUs, HETERO:GPU.0,GPU.1 + MODEL_DISTRIBUTION_POLICY=PIPELINE_PARALLEL silently places the entire model on the first GPU whenever it fits there. The second GPU gets nothing. No error or warning is emitted, and the server reports the model as AVAILABLE, so the user believes pipeline parallelism is active when it is not.

Hardware: 2x Intel Arc Pro B70 (32 GB each), driver 32.0.101.8992.

Measured after load (Windows \GPU Adapter Memory(*)\Dedicated Usage):

Generation works, but only because nothing ever crosses devices. This is effectively single-GPU execution.

Cause (source, tag 2026.4.0, src/plugins/hetero/src/plugin.cpp):

354: if (available_device_mem_map[device_name] >= 1.2 * total_ops_size || device_name.find("CPU") == 0) {
355:     device_config[ov::internal::query_model_ratio.name()] = 1.0f;
...
374: available_device_mem_map.erase(device_name);

The first device in the priority list passes the "fits alone" check, gets query_model_ratio = 1.0, claims every op, and is erased from the map before the second device is considered. The ~26 GB int8 model fits within 1.2x of one 32 GB card, so the second card is never used even though PIPELINE_PARALLEL was explicitly requested.

I understand HETERO spilling only when needed may be intended for the default policy. But when PIPELINE_PARALLEL is explicitly set on multiple identical discrete GPUs, silently ignoring it is surprising. It makes it impossible to use a second card for extra KV-cache headroom (long context), which is the main reason to split a model that technically fits one card.

A fix for this exact logic was proposed in #34702 (proportional split by memory share when more than one discrete device is present). It was auto-closed as stale in July 2026 without review.

Requested: either honour an explicit PIPELINE_PARALLEL across all listed discrete devices (e.g. the #34702 approach), or expose a way to control the per-device split ratio. At minimum, log a warning when the policy results in a single-device placement.

Related (crashes once a split does happen, all on cards too small to hold the model alone): #37450 (2x A770, Qwen3.8-27B), #37692 (2x Arc Pro B50, Qwen3.8-27B), #37006 (2x A770).

Step-by-step reproduction

1. Download OpenVINO/Qwen3.8-27B-int8-ov to E:/models/Qwen3.8-27B-int8-ov.
2. Configure and start OVMS 2026.4.0 (Windows binary package, after setupvars.bat):

ovms.exe --configure --model_path "E:/models/Qwen3.8-27B-int8-ov" --task text_generation ^
  --target_device "HETERO:GPU.0,GPU.1" --model_distribution_policy PIPELINE_PARALLEL ^
  --draft_model_path "E:/models/Qwen3.8-27B-int8-ov" --enable_prefix_caching false --cache_size 0 --overwrite_models

ovms.exe --model_path "E:/models/Qwen3.8-27B-int8-ov" --model_name qwen3.8-27b --rest_port 8081

Generated graph.pbtxt node options:

device: "HETERO:GPU.0,GPU.1",
plugin_config: '{"MODEL_DISTRIBUTION_POLICY":"PIPELINE_PARALLEL"}',

1. Wait for state changed to: AVAILABLE.
2. Check per-GPU memory (PowerShell): Get-Counter '\GPU Adapter Memory(*)\Dedicated Usage'. One B70 holds ~29.8 GB, the other 0 GB.
3. POST /v3/chat/completions succeeds (~37 tok/s with MTP). This is single-GPU performance.

On an earlier OpenVINO 2026.4 dev build, OPENVINO_HETERO_VISUALIZE=1 confirmed a single DeviceSubgraph with device=GPU.0 covering the whole model.

Relevant log output

[servable_initializer.cpp:688] Initializing Visual Language Model Continuous Batching servable
[servable_initializer.cpp:239] Draft model strategy: MTP (Multi-Token Prediction)
[servable_initializer.cpp:231] Auto-detected tool_parser: qwen3coder
[servable_initializer.cpp:241] Auto-detected reasoning_parser: qwen3
[pipelinedefinitionstatus.hpp:61] Mediapipe: qwen3.8-27b state changed to: AVAILABLE after handling: ValidationPassedEvent:

PS> Get-Counter '\GPU Adapter Memory(*)\Dedicated Usage'
luid_0x00000000_0x0000d726  29.80 GB
luid_0x00000000_0x0000ea2b   0.00 GB