CUDA: binary classification probability saturation (prob→1.0) — non-deterministic bug in col-wise histogram kernel

Environment

Bug description

Binary classification with device="cuda" produces saturated predictions (prob_max=1.0000, hundreds of samples stuck at prob≈1.0) while CPU training on identical data/params gives healthy predictions (prob_max<0.65). AUC remains plausible (~0.80), which masks the bug in standard evaluation.

The bug is non-deterministic: ~3 out of 5 seeds trigger saturation. The same seed can give HEALTHY on one run and DEGENERATE on another.

DEGENERATE = prob_max > 0.98 OR n_samples_with_prob ≥ 1.0 > 10
HEALTHY = prob_max < 0.80 (expected for this task)

Reproduction

Trigger conditions (confirmed via 30-run diagnostic)

Key finding: force_col_wise=True is the most stable trigger. LightGBM auto-selects col-wise for large feature counts (380 features), explaining why the default also triggers it intermittently.

Seed sensitivity (device="cuda", 380 features, 1M–2M rows)

Same seed can produce different status across runs → race condition in CUDA reduction.

Hyperparameters used (champion model params, but bug is not param-specific)

params = {
    "objective":         "binary",
    "metric":            "auc",
    "device":            "cuda",
    "num_threads":       14,
    "seed":              42,
    "learning_rate":     0.009518840502449276,
    "num_leaves":        186,
    "max_depth":         7,
    "min_child_samples": 100,
    "feature_fraction":  0.5934692247305747,
    "bagging_fraction":  0.5179519471812114,
    "bagging_freq":      4,
    "lambda_l1":         0.5924602885530806,
    "lambda_l2":         0.1274528927681517,
    "min_gain_to_split": 0.001,
}
lgb.train(params, dtrain, num_boost_round=200)

Code to reproduce (requires dataset, see below)

import lightgbm as lgb
import numpy as np
import pandas as pd

df = pd.read_parquet("mre_minimal_1000k.parquet")
train = df[df["split"] == "train"]
val   = df[df["split"] == "val"]
feat_cols = [c for c in df.columns if c.startswith("f")]

X_tr, y_tr = train[feat_cols].to_numpy(dtype=np.float32), train["target"].to_numpy()
X_va, y_va = val[feat_cols].to_numpy(dtype=np.float32),   val["target"].to_numpy()

params = {
    "objective": "binary", "metric": "auc", "verbose": -1,
    "device": "cuda", "num_threads": 14, "seed": 42,
    "learning_rate": 0.009518840502449276, "num_leaves": 186, "max_depth": 7,
    "min_child_samples": 100, "feature_fraction": 0.5934692247305747,
    "bagging_fraction": 0.5179519471812114, "bagging_freq": 4,
    "lambda_l1": 0.5924602885530806, "lambda_l2": 0.1274528927681517,
    "min_gain_to_split": 0.001,
}

bst = lgb.train(params, lgb.Dataset(X_tr, label=y_tr), num_boost_round=200)
prob = bst.predict(X_va)

print(f"prob_max  = {prob.max():.6f}")          # Expected HEALTHY: < 0.70
print(f"[email protected]   = {(prob >= 0.999999).sum()}") # Expected HEALTHY: 0
# DEGENERATE output: prob_max=1.000000, [email protected]=646–1273

Dataset

1M rows × 380 features (float32, binary target, pos_rate≈0.09).
Derived from public Binance BTC/USDT 30-min OHLCV (2023-02 – 2024-12).
Feature columns renamed f0..f379 (no domain identifiers).
File: mre_minimal_1000k.parquet — 743 MB (too large for attachment).

Available on request — please comment and I will share via Google Drive.

What was ruled out

• Synthetic data (i.i.d. Gaussian, heavy-tail t(df=2), factor model, AR(1), mixed) — does NOT reproduce the bug across all tested sizes (200K–2M rows, 380 features). The bug requires real financial time-series data with temporal autocorrelation and specific gradient distributions.
• Row shuffling — destroys the bug. Temporal order of rows is required for reproduction, suggesting contiguous memory access patterns interact with the CUDA histogram kernel.
• max_bin — varying 31/63/127/255 does not reliably fix or trigger the bug.
• gpu_use_dp=True — does not fix the bug.
• Feature pre-filter — feature_pre_filter=False is NOT set; this is a different known issue.

Root cause hypothesis

The col-wise histogram CUDA kernel has a race condition in its parallel reduction of gradient/hessian sums. When many temporally-autocorrelated rows map to the same histogram bin, concurrent thread blocks accumulate incorrect (near-zero) hessian sums. This causes leaf values = -Σg / (Σh + λ) to blow up, pushing raw scores to ±∞ within the first few trees. Subsequent iterations cannot recover because hessians for saturated samples ≈ 0.

Evidence:

1. force_col_wise=True reliably triggers; force_row_wise=True mostly safe
2. Same seed gives HEALTHY/DEGENERATE on different runs (non-deterministic)
3. Row shuffling prevents the bug (destroys temporal bin concentration)
4. AUC stays ~0.80 (saturation doesn't hurt ranking, only threshold-based selection)

Workaround

Add a health-check after training and retry with a different seed:

prob = bst.predict(X_val)
if prob.max() > 0.98 or (prob >= 0.999999).sum() > 10:
    raise RuntimeError("CUDA saturation detected — retrain with different seed")

Or force CPU for binary tasks on this hardware:

params["device"] = "cpu"  # safe, but ~3× slower