[RFC] Leaf/Depth Based Feature Masks

Summary

Add optional depth-based feature allow-lists so users can restrict which features may split at each tree depth, while still learning split thresholds. The similar feature control is forcedsplits_filename, which forces both feature and threshold. The proposed method keeps learned splits, but limits the possible features by depth.

You can find my current approach within the fork at: https://github.com/riverleversee/LightGBM/tree/feature/depth-feature-constraints

I am hoping to get feedback on that implementation so I can improve it and if possibly useful to others submit it for a Pull Request.

Motivation

I originally was looking to improve the performance of models across regime boundaries. To do this I looked into mixture of expert methods as well as investigating the importance/gain of my regime terms. Given the split based nature of tree learners it seemed natural to first focus on splitting regime terms early in the tree and focus later splits on local regime behavior. In my view this would allow the LGBM to internally form a mixture of experts inspired division during early splits in every tree.

Description

Extend the existing forced-splits JSON with an optional depth_feature_constraints array. Each entry specifies a min_depth and a non-empty features allow-list. At leaf depth d, the stage with the greatest min_depth <= d (if any) is active: only those features may be chosen, while split thresholds remain learned. A later stage fully replaces the earlier allow-list (stages are not merged), so feature sets can be specified layer by layer. Depths with no matching stage are unconstrained by this mechanism.

Composition with existing feature gating

• Active depth stage: the stage allow-list is a candidate mask. It bypasses feature_fraction / feature_fraction_bynode, then intersects interaction_constraints for the current branch.
• No matching stage: normal ColSampler behavior applies (feature_fraction / feature_fraction_bynode, then interaction_constraints).
• Classic forced splits: unchanged when depth_feature_constraints is absent. Both may appear in the same JSON: classic feature/threshold nodes still force the prefix path; depth stages gate candidate features during subsequent best-first growth.

My biggest current concern here is lack of overlap possible with feature fraction. This could probably be changed so a user ccan choose to apply feature fraction to the surviving features at each depth.

Valid only for CPU + tree_learner=serial + use_quantized_grad=false.

References

I have tried to make sure as few edits are in existing files as possible, and everything falls behind the existing forcedsplits as to add as little overhead as possible.

Here is an already posted issue I think this would help address: https://github.com/lightgbm-org/LightGBM/issues/5224 That issue is slightly narrower in scope as my approach adds depth level constraints which also allow you to deny splits on variables you expect to be broad or nonspecific (i.e. help you prevent memorization using slow varying terms by denying deep splits access to those terms).