Baike.dev
Log in
Back to tool/Back to issues
#38246·openvino

[Bug]: ROI tensor keeps its owner's strides after set_shape() expands the rank — ITensor::copy_to() then reads out of bounds (SIGSEGV in production)

Author: SyueYiLiaoCreated Sep 18, 2026Updated Sep 18, 2026
Labelsbugsupport_request

OpenVINO Version

2026.5.0-23099-d015add94c1

Operating System

Other (Please specify in description)

Device used for inference

GPU

Framework

None

Model used

OpenVINO/gemma-4-26b-a4b-it-int4-ov

Issue description

System information
OpenVINO version 2026.5.0-23099-d015add94c1 (stock binary; also reproduced with a self-build of the same commit)
Still present on releases/2026/4 (9bcdbd5e, 2026-09-17) and master (2d4c4804, 2026-09-18) — make_tensor.cpp and itensor.cpp are byte-identical to the build we crashed on, and the patch below applies cleanly to both (git apply --check verified on each)
Consumer OpenVINO Model Server 2026.5.0 (openvino/model_server:weekly, digest sha256:fcc16047f413…), GenAI 2026.5.0.0-3455-a521ae75728
OS / compiler Ubuntu 24.04, gcc 13
Hardware Intel Core Ultra 7 356H (Panther Lake) + Arc Pro B70 32 GB, driver xe, kernel 6.6.129
Model OpenVINO/gemma-4-26b-a4b-it-int4-ov served through the OVMS VLM pipeline

Summary

RoiTensor::get_strides() returns the strides of the tensor it is a view of, but BaseRoiTensor::set_shape() is allowed to expand the ROI's rank with leading ones (added in #22257). After such a set_shape(), the tensor reports a shape of rank N and strides of rank M < N.

ITensor::copy_to() performs that set_shape() itself and then walks shape and strides in lockstep, so a rank-1 ROI used as the destination of a rank-2 copy makes it read dst_strides[1] out of bounds. When the stale value there happens to compare equal, the "find a step" loop never fires, both stride vectors are left empty, and the copy loop evaluates src_strides[src_strides.size() - 1] — that is, src_strides[SIZE_MAX] — which dereferences data() - 8 on an empty vector and faults at 0xFFFFFFFFFFFFFFF8.

This kills the whole model server process. We hit it 9 times across independent runs on a stock image.

Minimal reproducer (no GenAI, no model server, no model)

cpp
#include <openvino/openvino.hpp>
#include <iostream>

static void dump(const char* when, const ov::Tensor& t) {
    std::cout << when << ": shape " << t.get_shape() << " (rank " << t.get_shape().size()
              << "), strides " << t.get_strides() << " (rank " << t.get_strides().size() << ")\n";
}

int main() {
    ov::Tensor owner(ov::element::i64, ov::Shape{4});                 // rank 1
    ov::Tensor dst_roi(owner, ov::Coordinate{0}, ov::Coordinate{1});  // rank-1 ROI, 1 element
    ov::Tensor src(ov::element::i64, ov::Shape{1, 1});                // rank 2, 1 element
    src.data<int64_t>()[0] = 42;

    dump("before", dst_roi);
    src.copy_to(dst_roi);      // copy_to() calls dst->set_shape({1,1}) first
    dump("after ", dst_roi);

    std::cout << (dst_roi.get_shape().size() == dst_roi.get_strides().size()
                  ? "OK: shape and strides have the same rank\n"
                  : "BUG: shape and strides have different ranks\n");
}
before: shape [1] (rank 1), strides Strides{8} (rank 1)
after : shape [1,1] (rank 2), strides Strides{8} (rank 1)      <-- inconsistent
BUG: shape and strides have different ranks

The invariant violation reproduces 100% of the time, on the 2026.5 build above and on the current releases/2026/4 and master sources (those two files are unchanged there). The out-of-bounds read it causes is reported deterministically by valgrind, so you do not need to reproduce the crash itself to see the defect:

==336== Invalid read of size 8
==336==    at 0x507C73F: ov::ITensor::copy_to(std::shared_ptr<ov::ITensor> const&) const (in /ovms/lib/libopenvino.so.2026.5.0)
==336==    by 0x50782DF: ov::Tensor::copy_to(ov::Tensor) const (in /ovms/lib/libopenvino.so.2026.5.0)
==336==    by 0x10A781: main (/roi_repro.cpp:18)
==336==  Address 0x618c7f8 is 0 bytes after a block of size 8 alloc'd
==336==    at 0x4846FA3: operator new(unsigned long) (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==336==    by 0x4C94226: ov::Strides::Strides(std::initializer_list<unsigned long> const&) (in /ovms/lib/libopenvino.so.2026.5.0)
==336==    by 0x507C0F8: ov::ITensor::copy_to(std::shared_ptr<ov::ITensor> const&) const (in /ovms/lib/libopenvino.so.2026.5.0)
==336==    by 0x50782DF: ov::Tensor::copy_to(ov::Tensor) const (in /ovms/lib/libopenvino.so.2026.5.0)
==336==    by 0x10A781: main (/roi_repro.cpp:18)
==336== ERROR SUMMARY: 2 errors from 2 contexts (suppressed: 0 from 0)

Whether the process actually dies depends on what that out-of-bounds read returns, which is why the production failure looks random while always landing on the same instruction.

Root cause

src/inference/src/dev/make_tensor.cpp:

cpp
void BaseRoiTensor::set_shape(ov::Shape new_shape) {       // :427
    OPENVINO_ASSERT(new_shape.size() >= m_shape.size());   // :428 rank may grow
    ...                                                     // the added dimensions must be 1
    m_shape = std::move(new_shape);
}

const Strides& RoiTensor::get_strides() const {            // :477 (RoiRemoteTensor :527 is identical)
    return m_owner->get_strides();                         // still the owner's rank
}

src/core/src/runtime/itensor.cpp:

cpp
const auto& shape = get_shape();
if (shape != dst->get_shape())
    dst->set_shape(shape);                                  // :84  dst's shape rank grows here
...
src_strides = get_strides();                                // :110
dst_strides = dst->get_strides();                           // :111 rank does NOT grow
ov::Strides src_str, dst_str;                               // :113 left empty unless a step is found
for (size_t inverted_idx = shape_rank - 1; inverted_idx < shape_rank; --inverted_idx) {
    if (default_strides[inverted_idx] == src_strides[inverted_idx] &&      // :119
        src_strides[inverted_idx] == dst_strides[inverted_idx]) {          // :120 OOB read when ranks differ
        continue;
    }
    ... // only here are src_str/dst_str resized
}
src_strides = std::move(src_str);                           // :150 stays empty if nothing above fired
...
copy_function(src_data + src_idx, dst_data + dst_idx,
              src_strides[src_strides.size() - 1]);         // :174 SIZE_MAX index on an empty vector

How it shows up in a real service

OVMS serving a VLM model reaches this on every decode step: ModelRunner::forward() (openvino.genai, continuous_batching/model_runner.hpp:804) copies a per-token position_ids element of shape {1,1} into a ROI of the rank-1 position_ids model input.

The run below used only synthetic requests (a fictional transit-operations assistant): one streaming tool-calling chat request that enters a repetition loop and is cut off by the client after 20 s, followed by one unary /v3/responses request. The server died on the 16th such cycle. The servable was configured with no explicit parser, so OVMS auto-detected gemma4 for both the tool and reasoning parsers — the recommended configuration for this model. The core is the stock binary; only GenAI was swapped for a self-build of the same commit so that frames carry file and line numbers. The first crash of this investigation happened on a completely unmodified image, with no self-built component at all — the GenAI swap only made the frames readable, it did not create the fault.

[2026-09-18 04:16:41][serving][info][server.cpp:116] OpenVINO backend 2026.5.0-23099-d015add94c1
[2026-09-18 04:16:41][llm_calculator][info][servable_initializer.cpp:231] Auto-detected tool_parser: gemma4
[2026-09-18 04:16:41][llm_calculator][info][servable_initializer.cpp:241] Auto-detected reasoning_parser: gemma4
[2026-09-18 04:18:51.001][157][llm_executor][info][legacy_executor.cpp:76] All requests: 1;

Thread 140 "ovms" received signal SIGSEGV, Segmentation fault.
0x00007ffff2779275 in ov::ITensor::copy_to(std::shared_ptr<ov::ITensor> const&) const

#0  ov::ITensor::copy_to(std::shared_ptr<ov::ITensor> const&) const     libopenvino.so.2650
#1  ov::Tensor::copy_to(ov::Tensor) const                               libopenvino.so.2650
#2  ov::genai::ModelRunner::forward(...)                     continuous_batching/model_runner.hpp:804
#3  ov::genai::ContinuousBatchingPipeline::ContinuousBatchingImpl::step(...)
#4  ov::genai::ContinuousBatchingPipeline::ContinuousBatchingImpl::generate(...)  pipeline_impl.cpp:766
#5  ov::genai::ContinuousBatchingPipeline::IContinuousBatchingPipeline::generate(...) pipeline_base.cpp:482
#6  ov::genai::ContinuousBatchingPipeline::IContinuousBatchingPipeline::generate(...) pipeline_base.cpp:581
#7  ov::genai::ContinuousBatchingPipeline::generate(...)                 pipeline.cpp:668
#8  ov::genai::ContinuousBatchingPipeline::generate<...>(...)            continuous_batching_pipeline.hpp:337
#9  ov::genai::VLMPipeline::VLMContinuousBatchingAdapter::generate(...)  continuous_batching_adapter.hpp:93
#12 ov::genai::VLMPipeline::generate(...)                                visual_language/pipeline.cpp:1051
#13 ovms::VisualLanguageModelLegacyExecutor::processRequest()
#14 ovms::VisualLanguageModelLegacyExecutorWrapper::run(...)

The tensors at the faulting call, read out of the inferior at the stop:

shape strides
src (position_ids_elem) {1, 1} {8, 8}
dst (the ROI, after copy_to's set_shape) {1, 1} {8}
dst's owner (position_ids model input) {1} {8}
ROI coordinates begin {0}, end {1} element type i64

The faulting instruction is mov rax,[rax-0x8] with rax = 0 — an empty std::vector indexed with size() - 1. The kernel logs segfault at fffffffffffffff8 ... in libopenvino.so.2026.5.0. Seven crashes across independent runs all landed on the same instruction (ip - vma_base identical every time) but at unpredictable request counts (after 1, 1, 7, 8, 10, 16, 16, 27, 27 cycles), which is what you would expect when an out-of-bounds read decides the outcome. The full gdb output is attached.

Suggested fix

Keep a ROI's strides at the same rank as its shape. The expanded dimensions are ones, so their stride is the row-major continuation of the dimension below. RoiTensor and RoiRemoteTensor have the same mismatch, so the helper lives in their shared base:

diff
diff --git a/src/inference/src/dev/make_tensor.cpp b/src/inference/src/dev/make_tensor.cpp
index 9ef2032db1..ba9bca2e24 100644
--- a/src/inference/src/dev/make_tensor.cpp
+++ b/src/inference/src/dev/make_tensor.cpp
@@ -448,6 +448,7 @@ public:
             std::distance(new_shape.cbegin(), new_dim.base()) - 1);
 
         m_shape = std::move(new_shape);
+        update_padded_strides();
     }
 
     size_t get_offset() const {
@@ -455,10 +456,36 @@ public:
     }
 
 protected:
+    // set_shape() above may expand the rank of the ROI with leading ones, while the strides of a
+    // ROI are the owner's and keep the owner's rank. A tensor whose shape and strides have
+    // different ranks breaks every consumer that walks the two in lockstep -- ITensor::copy_to()
+    // reads dst_strides[i] past the end of the vector -- so keep a padded copy to hand out
+    // instead. The expanding dimensions are ones (set_shape() enforces that), so their stride is
+    // whatever keeps the row-major relation with the dimension below.
+    void update_padded_strides() {
+        const auto& owner_strides = m_owner->get_strides();
+        if (m_shape.size() <= owner_strides.size()) {
+            m_padded_strides.clear();
+            return;
+        }
+        const auto pad = m_shape.size() - owner_strides.size();
+        m_padded_strides.assign(m_shape.size(), 0);
+        std::copy(owner_strides.begin(), owner_strides.end(), m_padded_strides.begin() + pad);
+        for (auto i = pad; i-- > 0;) {
+            m_padded_strides[i] = m_shape[i + 1] * m_padded_strides[i + 1];
+        }
+    }
+
+    const Strides& roi_strides() const {
+        const auto& owner_strides = m_owner->get_strides();
+        return m_padded_strides.size() == m_shape.size() ? m_padded_strides : owner_strides;
+    }
+
     std::shared_ptr<ITensor> m_owner;
     Shape m_shape;
     const Shape m_capacity;
     const size_t m_offset;
+    Strides m_padded_strides;
 };
 
 /**
@@ -475,7 +502,7 @@ public:
     }
 
     const Strides& get_strides() const override {
-        return m_owner->get_strides();
+        return roi_strides();
     }
 
     const Shape& get_shape() const override {
@@ -525,7 +552,7 @@ public:
     }
 
     const Strides& get_strides() const override {
-        return m_owner->get_strides();
+        return roi_strides();
     }
 
     const Shape& get_shape() const override {

That is the whole change: 29 added / 2 removed lines in one file, against d015add94c1. It applies cleanly to master and to releases/2026/4 as well.

With it applied the reproducer prints strides Strides{8, 8} (rank 2), valgrind reports 0 errors, and copy_to() takes the fast memcpy path — the strides now equal the destination's and the source is contiguous — instead of the strided walk.

Worth hardening either way: itensor.cpp:174 indexes src_strides[src_strides.size() - 1] without checking that the vector is non-empty, and the loop at :118 indexes dst_strides using the shape's rank. An assert that both stride vectors have the shape's rank, or a fallback to the contiguous path when no step is found, would turn any future instance of this class of bug into a diagnosable error rather than a wild read.

What we verified

  • Patched core built from d015add94c1 and dropped into the stock model server image — only libopenvino.so replaced; every plugin and GenAI is the stock binary.
  • Every symbol the 17 in-image consumers resolve against the core is still defined (0 missing).
  • Greedy-decoding output byte-identical to the stock build across 15 requests.
  • The workload that used to kill the server within ~13 request cycles on average survived 137 and 120 cycles in two independent runs, with no crash and no new segfault in dmesg. (Soak runs were driven under both parser configurations; the crash reproduces under both, so neither the failure nor the fix is configuration-specific.)

Note on the caller

The GenAI side has an inconsistency worth a look, though it is not the defect: for a rank-2 position_ids element ({1,1}), ModelRunner::forward() allocates the position_ids model input as rank-1 {total_num_tokens} (model_runner.hpp:573), while the 3-D (M-RoPE) branch keeps the original rank. That is what produces the rank-mismatched pair here. With the core fixed the usage becomes legal, so we have not filed it separately — happy to do so if you prefer.

ovms-vlm-segv-gdb.txt

Step-by-step reproduction

No response

Relevant log output

bash

Issue submission checklist

  • I'm reporting an issue. It's not a question.
  • I checked the problem with the documentation, FAQ, open issues, Stack Overflow, etc., and have not found a solution.
  • There is reproducer code and related data files such as images, videos, models, etc.

Source: openvinotoolkit/openvino

View original on GitHubView discussion on GitHub