Hierarchical shrinkage (HTS): improving the accuracy and interpretability of tree-based methods

Abhineet Agarwal*, Yan Shuo Tan*, Omer Ronen, Chandan Singh*, Bin Yu

📄 Paper, 🗂 Doc, 📌 Citation

Hierarchical shrinkage is an extremely fast post-hoc regularization method which works on any decision tree (or tree-based ensemble, such as Random Forest). It does not modify the tree structure, and instead regularizes the tree by shrinking the prediction over each node towards the sample means of its ancestors (using a single regularization parameter). Experiments over a wide variety of datasets show that hierarchical shrinkage substantially increases the predictive performance of individual decision trees and decision-tree ensembles.

See some examples of how hierarchical shrinkage works on one-dimensional functions:

Step function

Linear data