feature_types silently discarded in certain situations

We ran into an issue recently with xgboost 3.2.0 where we ran incremental learning atop an existing xgboost model and the type information for the existing columns was lost / ignored. This created problems for our categorical columns.

Claude wrote up this bug report.

Training continuation silently discards feature_types when the base model holds a

valueless Categories container — and CPU/GPU disagree on the outcome

Version: 3.2.0 · Impact: silent training of a wrong model (GPU) / spurious hard error (CPU)

Summary

A CatContainer can be allocated but valueless — one column per feature, zero total categories:

"cats": {"enc": [{"offsets": [], "values": []}, ...N], "feature_segments": [0, ...N+1], "sorted_idx": []}

We have a production booster in exactly this state (many columns, no categories), produced by a multi-GPU cuDF training run. We have not isolated which step allocates it, and we would value a maintainer's read on that — see "Open question" below. The defect reported here is what xgboost does given a container in that state, which is reproducible by constructing one directly.

Two different predicates disagree about whether such a container holds categories:

The container above is non-empty by the first definition and has no categories by the second.

The chain

xgboost/sklearn.py:617 get_model_categories (called from dask/data.py:290 _extract_data and from XGBModel.fit at sklearn.py:1340/1784/2286) uses the column-count definition:

categories = model.get_categories()
if not categories.empty():
    # override the `feature_types`.
    return model, categories

So the user's ["c", "q", …] list is discarded and replaced by an object carrying no categories. _data_utils.py:731 get_ref_categories then sets feature_types = None, and data.py::_transform_cudf_df / _transform_pandas_df fall through to dtype inference. With float32 code columns, every slot is inferred numeric.

What happens next depends on which adapter you are on:

• GPU (CudfAdapter) — HasRefCategorical() is n_total_cats != 0 → false → the recode in proxy_dmatrix.cuh:35 is skipped and the batch is used as-is. Training completes and produces a booster whose trees split the declared-categorical slots as ordinals. The booster still reports feature_types == ["c", …], inherited from the base by core.py:3384 _assign_dmatrix_features. The model is silently wrong; the first symptom is a much later tree_model.cc:127 CHECK failure in get_dump (see report 01 for that message being inverted).
• CPU (ColumnarAdapter) — HasRefCategorical() is !ref_cats_.Empty() → true → Recode → BasicChecks compares orig_enc.Size() (N) with new_enc.Size() (0) and aborts with cat_container.h:29: New and old encoding should have the same number of columns.

Same input, same defect, two incompatible outcomes — neither of which is "use the feature_types the caller passed".

Reproduction (CPU, no GPU required)

import json, numpy as np, pandas as pd, xgboost as xgb
from xgboost.sklearn import get_model_categories
from xgboost._data_utils import Categories

names, ftypes = ["cat_a", "num_b", "cat_c"], ["c", "q", "c"]
rng = np.random.default_rng(0)
pdf = pd.DataFrame({
    "cat_a": rng.integers(0, 12, 3000).astype("float32"),
    "num_b": rng.normal(size=3000).astype("float32"),
    "cat_c": rng.integers(0, 7, 3000).astype("float32"),
})
y = (pdf.cat_a < 4).to_numpy().astype("float32")
x = pdf.to_numpy(dtype="float32")

dm = xgb.QuantileDMatrix(x, label=y, feature_names=names, feature_types=ftypes, enable_categorical=True)
base = xgb.train({"objective": "binary:logistic", "tree_method": "hist"}, dm, num_boost_round=25)

# What a cuDF/GPU run serializes: N columns, zero categories. (A numpy run writes {"enc": [], ...}.)
raw = json.loads(base.save_raw("json").decode())
raw["learner"]["gradient_booster"]["model"]["cats"] = {
    "enc": [{"offsets": [], "values": []} for _ in names],
    "feature_segments": [0] * (len(names) + 1),
    "sorted_idx": [],
}
b = xgb.Booster(); b.load_model(bytearray(json.dumps(raw), "utf-8"))

print(b.get_categories().empty())                             # False  <- but it holds no categories
print(type(get_model_categories(pdf, b, ftypes)[1]).__name__)  # Categories  <- ftypes discarded
print(get_model_categories(x,   b, ftypes)[1])                 # ['c','q','c'] <- numpy escapes

_, hijacked = get_model_categories(pdf, b, ftypes)
xgb.QuantileDMatrix(pdf, label=y, feature_names=names, feature_types=hijacked, enable_categorical=True)
# XGBoostError: cat_container.h:29: New and old encoding should have the same number of columns.

On a cuDF frame the last call does not raise; it builds an all-numeric matrix, and xgb.dask.train(..., xgb_model=b) then appends numeric-split trees to a categorical base. We hit this in production: 2000 warm-start rounds over 26 "c" slots produced 1748 numeric splits and zero categorical splits, in a model whose base had 14088 categorical split nodes and no numeric ones.

Suggested fixes (any one of these breaks the chain)

1. Make "empty" mean "holds no categories." CatContainer::Empty() returning columns.empty() || n_total_cats == 0 would make get_model_categories fall through to the caller's feature_types, which is the correct behaviour here. This also aligns ColumnarAdapter::HasRefCategorical() with CudfAdapter::HasRefCategorical().
2. Do not allocate a per-column container when there are no categories — whatever path produces the allocated-but-valueless state should serialize {"enc": [], "feature_segments": [], "sorted_idx": []} instead, as the CPU paths already do. (Conditional on the open question below; we cannot name the producing code path.)
3. Do not discard feature_types for nothing. In get_model_categories, fall back to the caller's feature_types when categories carries no actual categories, rather than overriding with an object that carries none.
4. Independently: reconcile ColumnarAdapter::HasRefCategorical() and CudfAdapter::HasRefCategorical(). Whatever the intended semantics, a reference container that hard-errors on CPU and is silently ignored on GPU is a bug on one side or the other.

Fix 3 alone leaves fix 4's CPU/GPU divergence reachable by other routes; fix 1 or 2 is the durable one.

Open question

Which code path allocates a per-column CatContainer with zero categories? We observe it in a booster produced by xgboost.dask.train over cuDF frames of float32 code columns with feature_types=["c", …], on xgboost 3.2.0. We could not reproduce the allocation from any CPU input (see the note in the repro), so we cannot point at a specific line, and fix 2 above is contingent on this. Fixes 1, 3 and 4 stand regardless — they are about what happens once such a container exists, which is demonstrated above.