[CUDA] Training crashes when a categorical feature has more than 256 categories (illegal memory access / invalid argument)

Description

Training with device_type="cuda" fails hard whenever a categorical feature has more than 256 distinct categories. The boundary is exact and reproducible: 256 categories trains fine, 257 fails, on the same data and parameters. The same script with device_type="cpu" trains correctly at any cardinality.

The failure surfaces in two ways depending on parameters:

• With default threading, it raises a catchable LightGBMError:

  复制

  [LightGBM] [Fatal] [CUDA] invalid argument .../src/treelearner/cuda/cuda_single_gpu_tree_learner.cu 238

• With num_threads and/or max_cat_to_onehot set explicitly, it is an illegal memory access followed by std::terminate (the process aborts, so it cannot even be caught):

  复制

  [LightGBM] [Fatal] [CUDA] an illegal memory access was encountered .../src/treelearner/cuda/cuda_best_split_finder.cu 1811
  [LightGBM] [Fatal] [CUDA] an illegal memory access was encountered .../src/io/cuda/cuda_tree.cpp 38
  terminate called after throwing an instance of 'std::runtime_error'

Reproduced identically on two very different machines (x86_64 + RTX 3090 / sm_86, and aarch64 + GB10 / sm_121), so it does not look architecture-specific. The exact 256→257 boundary suggests an 8-bit limit somewhere in the CUDA categorical split path.

max_cat_threshold=8 does not avoid the crash. (The OpenCL device_type="gpu" backend rejects high-cardinality categoricals gracefully with "bin size … cannot run on GPU" — see #3106; the CUDA backend crashes instead.)

Reproducible example

import numpy as np
import pandas as pd
import lightgbm as lgb

rng = np.random.default_rng(0)
n = 200_000

for cardinality in (256, 257):
    df = pd.DataFrame({
        "x1": rng.normal(size=n),
        "cat": pd.Categorical(rng.integers(0, cardinality, n).astype(str)),
    })
    y = 0.5 * df["x1"] + (df["cat"].cat.codes % 7 - 3) * 0.05 + rng.normal(size=n)
    params = {
        "objective": "regression",
        "device_type": "cuda",
        "num_leaves": 31,
        "learning_rate": 0.05,
        "seed": 42,
        "verbosity": -1,
    }
    ds = lgb.Dataset(df, label=y, categorical_feature=["cat"])
    print(f"cardinality {cardinality}: ", end="", flush=True)
    booster = lgb.train(params, ds, num_boost_round=30)
    print("trained OK")

Output:

cardinality 256: trained OK
cardinality 257: [LightGBM] [Fatal] [CUDA] invalid argument .../src/treelearner/cuda/cuda_single_gpu_tree_learner.cu 238

Adding "num_threads": 8, "max_cat_to_onehot": 16 to params turns the same 257-category failure into the uncatchable illegal-memory-access abort in cuda_best_split_finder.cu:1811.

Environment info

LightGBM version or commit hash: 4.7.0 (PyPI sdist)

Command(s) you used to install LightGBM

# machine A (x86_64, RTX 3090)
pip install --no-binary lightgbm \
  --config-settings=cmake.define.USE_CUDA=ON \
  --config-settings=cmake.define.CMAKE_CUDA_ARCHITECTURES='86' \
  'lightgbm == 4.7.0'

# machine B (aarch64, NVIDIA GB10)
pip install --no-binary lightgbm \
  --config-settings=cmake.define.USE_CUDA=ON \
  --config-settings=cmake.define.CMAKE_CUDA_ARCHITECTURES='121-real;121-virtual' \
  'lightgbm == 4.7.0'

Both machines: Ubuntu 24.04.4 LTS, CUDA toolkit 13.0 (V13.0.88), NVIDIA driver 580.173.02, Python 3.14.6, numpy 2.x, pandas 3.x.

• Machine A: x86_64, GeForce RTX 3090 (sm_86)
• Machine B: aarch64, GB10 / DGX Spark (sm_121)

Additional Comments

The CPU device trains the identical Dataset/params without issue. The crash also reproduces with the categorical supplied as integer codes rather than pandas category dtype (same boundary).