#20278·tvm

[错误][CUDA] 具有前导零扩展的张量编译成功,但生成的内核在第一次 StreamSync 时会出现 cudaErrorIllegalAddress 错误

作者: lackjava-creator创建于 2026年9月6日更新于 2026年9月6日
标签type: bugneeds-triage

Environment

  • TVM version: main @ 48242ec33403f2b6e4fac6e763ca7a683fb9d5df (2026-09-03 21:20:55 -0400, reports 0.26.dev0); also present on release 0.26.0, 0.19.0 and 0.12.0.
  • Build/install: source build (/home/lxx/tvm-main), USE_CUDA=ON, nvcc 12.2, LLVM 15
  • OS/Python: Ubuntu 24.04.3 LTS, x86_64 (Xeon E5-2698 v4), Python 3.10
  • GPU/driver: Tesla V100-DGXS-32GB, driver 535.309.01, CUDA_VISIBLE_DEVICES=0, sm_70
  • Target: cuda. opt_level 3 and 0 both fail. llvm is clean.
  • Every run is one case per fresh processcudaErrorIllegalAddress is sticky for the life of a CUDA context.
  • Pipeline coverage (relax.get_pipeline("default_build") performs no operator fusion, so all three are reported separately). Verified 2026-09-06 on main @ 48242ec, CUDA, one pipeline per fresh process (a cudaErrorIllegalAddress poisons the context for the life of the process):
    • get_pipeline("default_build") (no FuseOps/FuseTIR) — fails, 2/2 fresh processes
    • get_pipeline("zero") (fuses) — fails, fresh process
    • forced FuseOps + FuseTIR appended to the legalize/fold sequence — fails, fresh process

Minimal reproducer

Single Neg node with input X : float32[0]. model.onnx + feed.npz + run.py attached (a MaxPool variant, model_maxpool_alt.onnx + feed_maxpool_alt.npz, is included to show the trigger is the shape, not the operator).

python
import numpy as np, onnx, tvm
from tvm import relax
from tvm.relax.frontend.onnx import from_onnx
model = onnx.load("model.onnx"); feed = dict(np.load("feed.npz"))     # X: float32[0]
mod = from_onnx(model, shape_dict={k: list(v.shape) for k, v in feed.items()}, keep_params_in_input=False)
order = [i.name for i in model.graph.input]
with tvm.transform.PassContext(opt_level=3):
    ex = tvm.compile(relax.get_pipeline("default_build")(mod), target="cuda")   # succeeds
dev = tvm.cuda(0)
out = relax.VirtualMachine(ex, dev)["main"](*[tvm.runtime.tensor(np.ascontiguousarray(feed[n]), dev) for n in order])
print(np.asarray(out.numpy()))                                                  # <-- faults here

Expected vs actual

  • Expected: an empty float32 tensor of shape (0,). numpy.negative(numpy.zeros((0,), numpy.float32)) is an empty array; onnxruntime returns the same; the identical program on TVM's llvm target returns the same.
  • Actual: tvm.compile succeeds and the VM call succeeds; the failure surfaces at the first synchronisation:
CUDA Runtime Error: cudaErrorIllegalAddress (700)
  cuda_device_api.cc:279  CUDADeviceAPI::StreamSync
  <- runtime/tensor.cc:104 Tensor::CopyToBytes

After this the whole CUDA context is poisoned; every later allocation in the same process fails.

  • Because the fault is asynchronous, a validator that never copies the empty result back to the host will report a pass. (This is why an earlier bisect of ours wrongly recorded a "(0.12, 0.19] regression" — the 0.12 run never …