apache-tvm-ffi<=0.1.12 (after #984) is incompatible with pinned tilelang==0.1.8 — measured working ceiling is 0.1.9
Summary
After #984 raised the bound to apache-tvm-ffi<=0.1.12 (pyproject.toml:21, setup.py:403), a fresh source install resolves tvm-ffi versions that are
incompatible with the pinned tilelang==0.1.8. We measured the full version matrix on a clean environment; only 0.1.9 and below work.
Version matrix (measured)
| apache-tvm-ffi | Result |
|---|---|
| 0.1.12 (current resolver pick) | import mamba_ssm crashes: AttributeError: attribute '__dict__' of 'type' objects is not writable in |
tvm_ffi/registry.py (_add_class_attrs). The import chain mamba_ssm/__init__ → modules.mamba3 → ops.tilelang… → import tilelang is unconditional, so the |
|
| whole package is unimportable. | |
| 0.1.11, 0.1.10 | Import OK, but every tilelang kernel compile fails: AttributeError: '_NestedLoopCheckVisitor' object has no attribute '_inst' |
(tilelang/analysis/nested_loop_checker.py, via tvm/runtime/support.py) — same failure as #907. All 90+ tests/ops/tilelang/test_mamba3_mimo.py tests |
|
| fail. | |
| 0.1.9, 0.1.8.post2 | Works. All tilelang forward-kernel tests pass on our box (remaining failures are an unrelated sm_121 shared-memory ceiling, |
| see note below). |
tilelang 0.1.8 itself declares apache-tvm-ffi>=0.1.2,~=0.1.0, so the resolver happily picks the broken versions unless mamba constrains them.
Suggestion
While tilelang==0.1.8 is pinned, the compatible constraint appears to be apache-tvm-ffi<0.1.10, not <=0.1.12. #984's motivation was flash-attn-4
wanting >=0.1.12 — with tilelang 0.1.8 pinned those two requirements look mutually exclusive, so this may need a tilelang bump rather than a bound tweak.
Related catch-22 we hit while validating: nvidia-cutlass-dsl 4.6.0 (the tests/ops/cute path) calls make_kwargs_wrapper(map_dataclass_to_tuple=…), which
only exists in tvm-ffi ≥0.1.10 — so no single tvm-ffi version currently satisfies both the tilelang and cute test paths.
Environment
- mamba @
f577286(source install,MAMBA_FORCE_BUILD=TRUE MAMBA_KEEP_CUDA_BUILD=TRUE pip install -e . --no-build-isolation) - Python 3.12.3, torch 2.12.0+cu130, CUDA 13.0
- NVIDIA GB10 (DGX Spark, sm_121, aarch64) — the 0.1.12 import crash was observed on this platform; we have not retested it on x86_64, though the 0.1.10/0.1.11 visitor failure is the same registry-layer bug as #907 (reported there on x86, so the drift is not platform-specific)
Repro
pip install "tilelang==0.1.8" "apache-tvm-ffi==0.1.12"
python -c "import tilelang" # AttributeError: attribute '__dict__' of 'type' objects is not writable
pip install "apache-tvm-ffi==0.1.11"
python -m pytest "tests/ops/tilelang/test_mamba3_mimo.py::test_fused_chunk_linear_attn_fwd_relative_error_lt_10pct[N64_P64_R4_C16_BB256]" -x
# AttributeError: '_NestedLoopCheckVisitor' object has no attribute '_inst'
pip install "apache-tvm-ffi==0.1.9"
python -m pytest "tests/ops/tilelang/test_mamba3_mimo.py::test_fused_chunk_linear_attn_fwd_relative_error_lt_10pct[N64_P64_R4_C16_BB256]" -x
# 1 passed (Separate observation, happy to file independently if useful: with 0.1.9 pinned, the Mamba-3 tilelang backward kernels still fail on GB10/sm_121 with
Failed to set the allowed dynamic shared memory size to 105120…227904 — they request 102–223 KB dynamic smem vs the 99 KB sharedMemPerBlockOptin cap on
this part; forward kernels all fit and pass. Related to what #955 was addressing for consumer GPUs.)
Source: state-spaces/mamba