RFC: Revisit Default Hyperparameters for Tree Learning in XGBoost

Related discussions: XGBoost issue #4986 (“Revisiting default parameter settings?”) and issue #6034 (“Add default ranges for hyperparameter tuning”). (github.com)

Summary

This RFC proposes a modest update to XGBoost’s default tree-learning configuration to improve out-of-the-box robustness and to better align the shipped defaults with published tunability results.

Current XGBoost parameter docs list eta=0.3, max_depth=6, min_child_weight=1, subsample=1, and colsample_bytree=1. The Python API docs still list num_boost_round=10 for both xgboost.train and xgboost.cv, while the current master branch source still sets DEFAULT_N_ESTIMATORS = 100 for the scikit-learn wrapper. (xgboost.readthedocs.io)

Proposed defaults

Parameter	Current	Proposed
eta	0.3	0.1
max_depth	6	10 6
min_child_weight	1	2
subsample	1.0	0.8
colsample_bytree	1.0	0.8
xgboost.train(..., num_boost_round=...)	10	300
xgboost.cv(..., num_boost_round=...)	10	300
sklearn fallback n_estimators	100	300

Parameters not changed

• gamma = 0
• colsample_bylevel = 1
• reg_lambda = 1
• reg_alpha = 0 (xgboost.readthedocs.io)

Motivation

The strongest academic evidence on XGBoost hyperparameter tunability remains Probst, Boulesteix, and Bischl (2019). Their cross-dataset study found that XGBoost has meaningful tunability under package defaults and materially lower tunability after replacing them with optimized defaults. The paper also states explicitly that, in XGBoost, the joint gain of tuning nrounds and eta is relatively low because the two are highly connected: when nrounds is set higher, eta should generally be set lower, and vice versa. (jmlr.org)

That same paper reports learned XGBoost defaults far from the current shipped defaults. The optimal-default values they report include very small eta, subsample below 1, colsample_bytree below 1, min_child_weight above 1, max_depth well above 6, and nrounds in the thousands rather than tens or hundreds. The exact reported values vary by metric, but the direction is consistent: smaller learning rate, larger round budget, moderate row/feature subsampling, somewhat deeper trees, and slightly more conservative leaf growth. (jmlr.org)

The broader boosting literature points in the same direction. Friedman’s original gradient boosting paper states that shrinkage introduces two coupled regularization parameters, the learning rate v and the number of components M, and that decreasing v increases the best value of M. XGBoost’s own parameter docs describe eta as step-size shrinkage that makes the process more conservative, and the Python API supports early_stopping_rounds directly. (cse.cuhk.edu.hk)

Proposal

1. Update core tree-learning defaults

Change the default tree-learning parameters to:

• eta = 0.1
• max_depth = 10
• min_child_weight = 2
• subsample = 0.8
• colsample_bytree = 0.8 (xgboost.readthedocs.io)

2. Align Python-facing training budgets

Set the Python-facing default training budgets to:

• xgboost.train(..., num_boost_round=300, ...)
• xgboost.cv(..., num_boost_round=300, ...)
• sklearn fallback DEFAULT_N_ESTIMATORS = 300 (xgboost.readthedocs.io)

This alignment removes the current discrepancy between the native Python learning API and the scikit-learn wrapper. There is little published justification for exposing a much smaller default training budget in one Python interface than in another when both ultimately control the same sequence of additive tree updates. The tunability literature instead argues that learning rate and training budget are tightly coupled and should be considered together. (jmlr.org)

3. Strengthen documentation around early stopping

Documentation and examples should more explicitly pair smaller eta with:

• larger round ceilings, and
• early stopping when validation data are available. (xgboost.readthedocs.io)

Rationale by parameter

eta: 0.3 → 0.1

The current docs list eta=0.3 and describe it as step-size shrinkage used to make boosting more conservative. Probst et al. report much smaller learned defaults for XGBoost than 0.3, and identify eta as one of the most tunable XGBoost parameters. Friedman’s shrinkage analysis also supports smaller learning rates when training budget permits. A move from 0.3 to 0.1 follows both the XGBoost documentation and the academic literature while avoiding a more disruptive jump to extremely small values. (xgboost.readthedocs.io)

max_depth: 6 → 10

The current docs list max_depth=6 and warn that deeper trees are more complex, more likely to overfit, and can consume memory aggressively. Probst et al. nevertheless report learned XGBoost defaults of 13, 14, and 11 across the metrics they study, all materially above the current default of 6. A default of 10 moves the shipped baseline closer to that region without fully matching the larger learned depths from the benchmark study. (xgboost.readthedocs.io)

min_child_weight: 1 → 2

The current docs list min_child_weight=1 and describe larger values as more conservative. Probst et al. report learned defaults for XGBoost in the neighborhood of 1.3 to 2.1, which supports a modest upward shift rather than a large one. A default of 2 is therefore consistent with both the documented semantics and the benchmark-based optimal defaults. (xgboost.readthedocs.io)

subsample: 1.0 → 0.8

The docs state that subsample is the row subsampling ratio and that subsampling helps prevent overfitting. Probst et al. report learned XGBoost defaults consistently below 1.0, roughly in the high-0.8 range depending on metric. A default of 0.8 is slightly more conservative than the learned defaults, but still very close to the region favored by the cross-dataset study. (xgboost.readthedocs.io)

colsample_bytree: 1.0 → 0.8

The docs state that colsample_bytree is the fraction of columns subsampled when constructing each tree, and that the colsample_by* parameters work cumulatively. Probst et al. report learned defaults for colsample_bytree around 0.71 to 0.75. A default of 0.8 moves clearly in the literature-supported direction while avoiding compounded regularization, since this RFC leaves colsample_bylevel unchanged. (xgboost.readthedocs.io)

Rationale for aligned default round counts

The strongest justification for increasing and aligning default round counts is internal consistency with the lower learning rate. The Python API currently defaults to num_boost_round=10, while the sklearn wrapper source still falls back to 100 estimators. At the same time, the literature says smaller learning rates require more iterations, and the XGBoost-specific tunability paper states directly that nrounds and eta are highly connected. (xgboost.readthedocs.io)

A default of 300 is justified as a practical aligned budget. Reducing eta from 0.3 to 0.1 lowers the per-round step size by a factor of three. Raising the sklearn fallback from 100 to 300 preserves that rough scaling. Aligning xgboost.train and xgboost.cv to the same 300-round default then removes the current API inconsistency. The literature supports the coupling between learning rate and rounds, even though it does not provide a universal exact scaling constant. (cse.cuhk.edu.hk)

Why other defaults are left unchanged

gamma, reg_lambda, and reg_alpha are left unchanged because the evidence for changing them is weaker and less consistent than the evidence for changing eta, training budget, moderate subsampling, min_child_weight, and depth. colsample_bylevel is left unchanged because the docs emphasize that the colsample_by* parameters are cumulative, so changing both colsample_bytree and colsample_bylevel simultaneously would stack the regularization. (xgboost.readthedocs.io)

Expected benefits

The expected effect is a more internally consistent and more robust out-of-the-box tree learner: smaller boosting steps, a larger and aligned default training budget, moderate row and feature randomness, a slightly higher child-weight threshold, and deeper baseline interaction capacity. This proposed default set is also materially closer to the regions favored by the cross-dataset tunability literature than the current documented defaults. (jmlr.org)

Risks and compatibility

The largest risk is increased training time from larger default training budgets. That increase is intentional: lowering eta without raising default round counts would leave the defaults internally inconsistent. A second risk is larger models and higher memory use, especially with max_depth=10; the parameter docs explicitly warn that deep trees can consume memory aggressively. A third risk is that enabling subsample and colsample_bytree below 1 introduces stochasticity into workflows that currently rely on fully deterministic default sampling settings. (xgboost.readthedocs.io)

Suggested release-note wording

Tree-learning defaults in XGBoost have been updated to improve robustness and internal consistency out of the box. The default learning rate was reduced, default depth and split conservatism were rebalanced, moderate row and feature subsampling were enabled by default, and Python-facing default training budgets were aligned upward to match the lower learning rate. Users comparing results across releases should account for the larger default training budget and should consider early stopping when validation data are available. (xgboost.readthedocs.io)

References

• Probst, Boulesteix, and Bischl (2019), Tunability: Importance of Hyperparameters of Machine Learning Algorithms. (jmlr.org)
• Friedman (2001), Greedy Function Approximation: A Gradient Boosting Machine. (cse.cuhk.edu.hk)
• XGBoost parameter documentation. (xgboost.readthedocs.io)
• XGBoost Python API documentation. (xgboost.readthedocs.io)
• XGBoost issue #4986. (github.com)
• XGBoost issue #6034. (github.com)