#3188·catboost

Add a parameter to set k-folds and train similarly to K-fold cross-validation for Ordered Boosting

Author: asdsadsaasd12312132Created Sep 7, 2026Updated Sep 14, 2026

The method used to create folds for Ordered Boosting has a massive flaw, which hurts performance. The problem is that the smallest folds have tiny sample sizes, and the gradients calculated from them are harmful/pointless to include since the sample size is typically too small for accurate predictions/gradients. For example, in a data set with 1000 samples and fold_len_multiplier=2, the tiny sample size folds occupy up to 80% of the total folds.

Fold sample size Cumulative % of total folds
1 10%
2 20%
4 30%
8 40%
16 50%
32 60%
64 70%
128 80%
256 90%
512 100%

Rather than exponentially increasing fold size with fold_len_multiplier, the folds should be generated using a similar method to K-fold cross-validation. For example, if K = 4, it would train like this

Training folds Testing fold
1 2
1 + 2 3
1 + 2 + 3 4

There should be a parameter to set K. This would allow us to choose an appropriate fold size to ensure there are sufficient samples for each fold.

This would also drastically reduce training time since in this example, the model only needs to be trained 3 times compared to 10 times in the first example.