Add a parameter to set k-folds and train similarly to K-fold cross-validation for Ordered Boosting
The method used to create folds for Ordered Boosting has a massive flaw, which hurts performance. The problem is that the smallest folds have tiny sample sizes, and the gradients calculated from them are harmful/pointless to include since the sample size is typically too small for accurate predictions/gradients. For example, in a data set with 1000 samples and fold_len_multiplier=2, the tiny sample size folds occupy up to 80% of the total folds.
| Fold sample size | Cumulative % of total folds |
|---|---|
| 1 | 10% |
| 2 | 20% |
| 4 | 30% |
| 8 | 40% |
| 16 | 50% |
| 32 | 60% |
| 64 | 70% |
| 128 | 80% |
| 256 | 90% |
| 512 | 100% |
Rather than exponentially increasing fold size with fold_len_multiplier, the folds should be generated using a similar method to K-fold cross-validation. For example, if K = 4, it would train like this
| Training folds | Testing fold |
|---|---|
| 1 | 2 |
| 1 + 2 | 3 |
| 1 + 2 + 3 | 4 |
There should be a parameter to set K. This would allow us to choose an appropriate fold size to ensure there are sufficient samples for each fold.
This would also drastically reduce training time since in this example, the model only needs to be trained 3 times compared to 10 times in the first example.
Source: catboost/catboost