#20279·tvm

[错误][CUDA][dlight] GPU 回退时,调用 sch.fuse 处理零循环范围,并以"除以零"错误终止

作者: lackjava-creator创建于 2026年9月6日更新于 2026年9月6日
标签type: bugneeds-triage

Environment

  • TVM version: main @ 48242ec33403f2b6e4fac6e763ca7a683fb9d5df (2026-09-03, reports 0.26.dev0); the same trigger fails on 0.26.0, and on 0.19.0 / 0.12.0 it surfaced as a ZeroDivisionError in Relay's pad_to_tensorcore.
  • Build / install: source build, USE_CUDA=ON, nvcc 12.2, LLVM 15
  • OS / Python: Ubuntu 24.04.3 LTS, x86_64, Python 3.10
  • GPU / driver: Tesla V100-DGXS-32GB, driver 535.309.01, sm_70
  • Target: cuda. opt_level 3 and 0 both fail. llvm is clean at both levels.
  • Pipeline coverage (relax.get_pipeline("default_build") performs no operator fusion, so all three are reported separately). Verified 2026-09-06 on 0.26.0, CUDA target — the abort happens at compile time, before any kernel launch:
    • get_pipeline("default_build") (no FuseOps/FuseTIR) — fails
    • get_pipeline("zero") (fuses) — fails
    • forced FuseOps + FuseTIRfails (also confirmed on main @ 48242ec at opt_level 3 and 0)

Minimal reproducer

Single Neg node with input X : float32[4,0] — a zero extent that is not on the leading axis. model.onnx + feed.npz + run.py attached.

python
import numpy as np, onnx, tvm
from tvm import relax
from tvm.relax.frontend.onnx import from_onnx
model = onnx.load("model.onnx"); feed = dict(np.load("feed.npz"))     # X: float32[4,0]
mod = from_onnx(model, shape_dict={k: list(v.shape) for k, v in feed.items()}, keep_params_in_input=False)
with tvm.transform.PassContext(opt_level=3):
    tvm.compile(relax.get_pipeline("default_build")(mod), target="cuda")   # <-- raises here

Companion note: the leading-axis zero-extent case does not fail here — it compiles and then faults at run time. The full matrix is: 0 on the leading axis (or a 1-D empty tensor) → runtime cudaErrorIllegalAddress; 0 on a non-leading axis → this compile-time abort.

Expected vs actual

  • Expected: compilation succeeds and the result is an empty float32 tensor of shape (4,0). numpy, onnxruntime, and TVM's own llvm target all produce that.
  • Actual:
InternalError: Check failed: pb->value != 0 (0 vs. 0) : Divide by zero

raised during tvm.compile, at both opt_level=3 and opt_level=0.

Root cause (if known)

dlight's GPU Fallback rule calls sch.fuse, and Fuse reconstructs the index of a fused loop as floordiv(floormod(fused, next_lower), lower). Here next_lower == 0, because it is the extent of the fused loop, which is 0. The zero is therefore the compiler's own loop extent, not a divisor coming from the model — an internal invariant that the scheduling rule breaks on itself. llvm is clean, so the rule is the producer. Never fixed. On 0.12/0.19 the same trigger hit pad_to_tensorcore in the Relay pipeline (ZeroDivisionError); the site moved in (0.19, main], the root cause did not.

Why this is a bug (not tolerance / not undefined behaviour)

A zero extent on a non-leading axis is legal in ONNX and defined by …