Classes

class HSTree (estimator_: sklearn.base.BaseEstimator = None, reg_param: float = 1, shrinkage_scheme_: str = 'node_based', max_leaf_nodes: int = None, random_state: int = None)

Base class for all estimators in scikit-learn.

Inheriting from this class provides default implementations of:

• setting and getting parameters used by GridSearchCV and friends;
• textual and HTML representation displayed in terminals and IDEs;
• estimator serialization;
• parameters validation;
• data validation;
• feature names validation.

Read more in the :ref:User Guide <rolling_your_own_estimator>.

Notes

All estimators should specify all the parameters that can be set at the class level in their __init__ as explicit keyword arguments (no *args or **kwargs).

Examples

>>> import numpy as np
>>> from sklearn.base import BaseEstimator
>>> class MyEstimator(BaseEstimator):
...     def __init__(self, *, param=1):
...         self.param = param
...     def fit(self, X, y=None):
...         self.is_fitted_ = True
...         return self
...     def predict(self, X):
...         return np.full(shape=X.shape[0], fill_value=self.param)
>>> estimator = MyEstimator(param=2)
>>> estimator.get_params()
{'param': 2}
>>> X = np.array([[1, 2], [2, 3], [3, 4]])
>>> y = np.array([1, 0, 1])
>>> estimator.fit(X, y).predict(X)
array([2, 2, 2])
>>> estimator.set_params(param=3).fit(X, y).predict(X)
array([3, 3, 3])

HSTree (Tree with hierarchical shrinkage applied). Hierarchical shinkage is an extremely fast post-hoc regularization method which works on any decision tree (or tree-based ensemble, such as Random Forest). It does not modify the tree structure, and instead regularizes the tree by shrinking the prediction over each node towards the sample means of its ancestors (using a single regularization parameter). Experiments over a wide variety of datasets show that hierarchical shrinkage substantially increases the predictive performance of individual decision trees and decision-tree ensembles. https://arxiv.org/abs/2202.00858

Params

estimator_: sklearn tree or tree ensemble model (e.g. RandomForest or GradientBoosting) Defaults to CART Classification Tree with 20 max leaf nodes Note: this estimator will be directly modified

reg_param: float Higher is more regularization (can be arbitrarily large, should not be < 0)

shrinkage_scheme: str Experimental: Used to experiment with different forms of shrinkage. options are: (i) node_based shrinks based on number of samples in parent node (ii) leaf_based only shrinks leaf nodes based on number of leaf samples (iii) constant shrinks every node by a constant lambda

max_leaf_nodes: int If estimator is None, then max_leaf_nodes is passed to the default decision tree

Expand source code

class HSTree(BaseEstimator):
    def __init__(
        self,
        estimator_: BaseEstimator = None,
        reg_param: float = 1,
        shrinkage_scheme_: str = "node_based",
        max_leaf_nodes: int = None,
        random_state: int = None,
    ):
        """HSTree (Tree with hierarchical shrinkage applied).
        Hierarchical shinkage is an extremely fast post-hoc regularization method which works on any decision tree (or tree-based ensemble, such as Random Forest).
        It does not modify the tree structure, and instead regularizes the tree by shrinking the prediction over each node towards the sample means of its ancestors (using a single regularization parameter).
        Experiments over a wide variety of datasets show that hierarchical shrinkage substantially increases the predictive performance of individual decision trees and decision-tree ensembles.
        https://arxiv.org/abs/2202.00858

        Params
        ------
        estimator_: sklearn tree or tree ensemble model (e.g. RandomForest or GradientBoosting)
            Defaults to CART Classification Tree with 20 max leaf nodes
            Note: this estimator will be directly modified

        reg_param: float
            Higher is more regularization (can be arbitrarily large, should not be < 0)

        shrinkage_scheme: str
            Experimental: Used to experiment with different forms of shrinkage. options are:
                (i) node_based shrinks based on number of samples in parent node
                (ii) leaf_based only shrinks leaf nodes based on number of leaf samples
                (iii) constant shrinks every node by a constant lambda

        max_leaf_nodes: int
            If estimator is None, then max_leaf_nodes is passed to the default decision tree
        """
        super().__init__()
        self.reg_param = reg_param
        # constructed here rather than defaulted in the signature, so that
        # separate models don't share (and refit) one estimator object
        if estimator_ is None:
            estimator_ = DecisionTreeClassifier(max_leaf_nodes=20)
        self.estimator_ = estimator_
        self.shrinkage_scheme_ = shrinkage_scheme_
        self.random_state = random_state
        if checks.check_is_fitted(self.estimator_):
            self._shrink()
        if max_leaf_nodes is not None:
            self.estimator_.max_leaf_nodes = max_leaf_nodes
            self.estimator_.random_state = random_state


    def get_rules(self, feature_names=None):
        """Return this model's rules as a DataFrame (see imodels.get_rules)."""
        from imodels.util.get_rules import get_rules
        return get_rules(self, feature_names=feature_names)

    def apply(self, X):
        """Return the leaf each sample reaches (see imodels.util.apply.apply_leaves)."""
        from imodels.util.apply import apply_leaves
        return apply_leaves(self, X)

    def get_params(self, deep=True):
        d = {
            "reg_param": self.reg_param,
            "estimator_": self.estimator_,
            "shrinkage_scheme_": self.shrinkage_scheme_,
            "max_leaf_nodes": self.estimator_.max_leaf_nodes,
        }
        if deep:
            return deepcopy(d)
        return d

    def fit(self, X, y, sample_weight=None, *args, **kwargs):
        # remove feature_names if it exists (note: only works as keyword-arg)
        # None returned if not passed
        feature_names = kwargs.pop("feature_names", None)
        # missing values are left to the wrapped estimator, which handles them for
        # sklearn decision trees and forests and raises for those that cannot
        X, y, feature_names = check_fit_arguments(
            self, X, y, feature_names, allow_nan=True)
        if feature_names is not None:
            self.feature_names = feature_names
        self.estimator_ = self.estimator_.fit(
            X, y, *args, sample_weight=sample_weight, **kwargs
        )
        self._shrink()

        # compute complexity
        if hasattr(self.estimator_, "tree_"):
            self.complexity_ = compute_tree_complexity(self.estimator_.tree_)
        elif hasattr(self.estimator_, "estimators_"):
            self.complexity_ = 0
            for i in range(len(self.estimator_.estimators_)):
                t = deepcopy(self.estimator_.estimators_[i])
                if isinstance(t, np.ndarray):
                    assert t.size == 1, "multiple trees stored under tree_?"
                    t = t[0]
                self.complexity_ += compute_tree_complexity(t.tree_)
        return self

    def _shrink_tree(
        self,
        tree,
        reg_param,
        i=0,
        parent_val=None,
        parent_num=None,
        cum_sum=0,
        values_normalized=None,
    ):
        """Shrink the tree"""
        if reg_param is None:
            reg_param = 1.0
        if values_normalized is None:
            # sklearn >= 1.3 stores classification tree values as probabilities
            # rather than counts, in which case they must not be renormalized.
            # Computed once at the root, since shrinking rewrites tree.value.
            values_normalized = _values_are_normalized(tree)
        left = tree.children_left[i]
        right = tree.children_right[i]
        is_leaf = left == right
        n_samples = tree.weighted_n_node_samples[i]
        if (
            isinstance(self, RegressorMixin)
            or isinstance(self.estimator_, GradientBoostingClassifier)
            or values_normalized
        ):
            val = deepcopy(tree.value[i, :, :])
        else:  # If classification, counts need normalizing into a probability vector
            val = tree.value[i, :, :] / n_samples

        # Step 1: Update cum_sum
        # if root
        if parent_val is None and parent_num is None:
            cum_sum = val

        # if has parent
        else:
            if self.shrinkage_scheme_ == "node_based":
                val_new = (val - parent_val) / (1 + reg_param / parent_num)
            elif self.shrinkage_scheme_ == "constant":
                val_new = (val - parent_val) / (1 + reg_param)
            else:  # leaf_based
                val_new = 0
            cum_sum += val_new

        # Step 2: Update node values
        if (
            self.shrinkage_scheme_ == "node_based"
            or self.shrinkage_scheme_ == "constant"
        ):
            tree.value[i, :, :] = cum_sum
        else:  # leaf_based
            if is_leaf:  # update node values if leaf_based
                root_val = tree.value[0, :, :]
                tree.value[i, :, :] = root_val + (val - root_val) / (
                    1 + reg_param / n_samples
                )
            else:
                tree.value[i, :, :] = val

                # Step 3: Recurse if not leaf
        if not is_leaf:
            self._shrink_tree(
                tree,
                reg_param,
                left,
                parent_val=val,
                parent_num=n_samples,
                cum_sum=deepcopy(cum_sum),
                values_normalized=values_normalized,
            )
            self._shrink_tree(
                tree,
                reg_param,
                right,
                parent_val=val,
                parent_num=n_samples,
                cum_sum=deepcopy(cum_sum),
                values_normalized=values_normalized,
            )

            # edit the non-leaf nodes for later visualization (doesn't effect predictions)

        return tree

    def _unwrap_tree(self, estimator):
        """Get the underlying sklearn tree, or None if there isn't one."""
        if isinstance(estimator, np.ndarray):
            assert estimator.size == 1, "multiple trees stored under tree_?"
            estimator = estimator[0]
        return getattr(estimator, "tree_", None)

    def _check_estimator_supported(self):
        """Reject estimators that shrinkage cannot be applied to.

        Shrinkage rewrites the node values of sklearn decision trees, so it needs
        either a fitted tree (tree_) or an ensemble of them (estimators_).
        Without this check an unsupported model is returned unchanged, which
        looks like it worked -- see https://github.com/csinva/imodels/issues/199
        """
        if hasattr(self.estimator_, "tree_"):
            return
        subestimators = getattr(self.estimator_, "estimators_", None)
        if subestimators is not None and len(subestimators) > 0:
            if self._unwrap_tree(subestimators[0]) is not None:
                return

        raise ValueError(
            f"{type(self.estimator_).__name__} is not supported by hierarchical "
            "shrinkage, which needs an estimator built from scikit-learn decision "
            "trees (exposing tree_, or estimators_ of trees) -- for example "
            "DecisionTreeClassifier, RandomForestRegressor or "
            "GradientBoostingClassifier. Applying shrinkage to it would leave the "
            "model unchanged."
        )

    def _shrink(self):
        self._check_estimator_supported()
        if hasattr(self.estimator_, "tree_"):
            self._shrink_tree(self.estimator_.tree_, self.reg_param)
        else:
            for t in self.estimator_.estimators_:
                self._shrink_tree(self._unwrap_tree(t), self.reg_param)

    def predict(self, X, *args, **kwargs):
        preds = self.estimator_.predict(X, *args, **kwargs)
        if hasattr(self.estimator_, "classes_"):
            return np.array([self.classes_[int(i)] for i in preds])
        else:
            return preds

    def predict_proba(self, X, *args, **kwargs):
        if hasattr(self.estimator_, "predict_proba"):
            probs = self.estimator_.predict_proba(X, *args, **kwargs)
            # the shrinkage arithmetic can leave values a hair outside [0, 1]
            # (e.g. -1e-17), which newer versions of sklearn's log_loss reject
            probs = np.clip(probs, 0, 1)
            totals = probs.sum(axis=1, keepdims=True)
            with np.errstate(invalid="ignore", divide="ignore"):
                normalized = probs / totals
            return np.where(totals > 0, normalized, 1 / probs.shape[1])
        else:
            return NotImplemented

    def __str__(self):
        # check if fitted
        if not checks.check_is_fitted(self.estimator_):
            s = self.__class__.__name__
            s += "("
            s += "est="
            s += repr(self.estimator_)
            s += ", "
            s += "reg_param="
            s += str(self.reg_param)
            s += ")"
            return s
        elif not hasattr(self.estimator_, "tree_"):
            # an ensemble: export_text only renders a single tree, so summarize
            n_trees = len(getattr(self.estimator_, "estimators_", []))
            s = "> ------------------------------\n"
            s += "> Tree ensemble with Hierarchical Shrinkage\n"
            s += "> \tPrediction is made by combining the predictions of each shrunk tree\n"
            s += "> ------------------------------" + "\n"
            s += f"> {type(self.estimator_).__name__} of {n_trees} trees, "
            s += f"reg_param={self.reg_param}\n"
            if hasattr(self, "complexity_"):
                s += f"> {self.complexity_} total non-leaf nodes\n"
            return s
        else:
            s = "> ------------------------------\n"
            s += "> Decision Tree with Hierarchical Shrinkage\n"
            s += "> \tPrediction is made by looking at the value in the appropriate leaf of the tree\n"
            s += "> ------------------------------" + "\n"

            if hasattr(self, "feature_names") and self.feature_names is not None:
                return s + export_text(
                    self.estimator_, feature_names=self.feature_names, show_weights=True
                )
            else:
                return s + export_text(self.estimator_, show_weights=True)

    def __repr__(self):
        # s = self.__class__.__name__
        # s += "("
        # s += "estimator_="
        # s += repr(self.estimator_)
        # s += ", "
        # s += "reg_param="
        # s += str(self.reg_param)
        # s += ", "
        # s += "shrinkage_scheme_="
        # s += self.shrinkage_scheme_
        # s += ")"
        # return s
        attr_list = ["estimator_", "reg_param", "shrinkage_scheme_"]
        s = self.__class__.__name__
        s += "("
        for attr in attr_list:
            s += attr + "=" + repr(getattr(self, attr)) + ", "
        s = s[:-2] + ")"
        return s

Ancestors

• sklearn.base.BaseEstimator
• sklearn.utils._repr_html.base.ReprHTMLMixin
• sklearn.utils._repr_html.base._HTMLDocumentationLinkMixin
• sklearn.utils._metadata_requests._MetadataRequester

Subclasses

• HSTreeClassifier
• HSTreeRegressor

Methods

def apply(self, X)

Return the leaf each sample reaches (see imodels.util.apply.apply_leaves).

Expand source code

def apply(self, X):
    """Return the leaf each sample reaches (see imodels.util.apply.apply_leaves)."""
    from imodels.util.apply import apply_leaves
    return apply_leaves(self, X)

def fit(self, X, y, sample_weight=None, *args, **kwargs)

Expand source code

def fit(self, X, y, sample_weight=None, *args, **kwargs):
    # remove feature_names if it exists (note: only works as keyword-arg)
    # None returned if not passed
    feature_names = kwargs.pop("feature_names", None)
    # missing values are left to the wrapped estimator, which handles them for
    # sklearn decision trees and forests and raises for those that cannot
    X, y, feature_names = check_fit_arguments(
        self, X, y, feature_names, allow_nan=True)
    if feature_names is not None:
        self.feature_names = feature_names
    self.estimator_ = self.estimator_.fit(
        X, y, *args, sample_weight=sample_weight, **kwargs
    )
    self._shrink()

    # compute complexity
    if hasattr(self.estimator_, "tree_"):
        self.complexity_ = compute_tree_complexity(self.estimator_.tree_)
    elif hasattr(self.estimator_, "estimators_"):
        self.complexity_ = 0
        for i in range(len(self.estimator_.estimators_)):
            t = deepcopy(self.estimator_.estimators_[i])
            if isinstance(t, np.ndarray):
                assert t.size == 1, "multiple trees stored under tree_?"
                t = t[0]
            self.complexity_ += compute_tree_complexity(t.tree_)
    return self

def get_params(self, deep=True)

Get parameters for this estimator.

Parameters

deep : bool, default=True

If True, will return the parameters for this estimator and contained subobjects that are estimators.

Returns

params : dict

Parameter names mapped to their values.

Expand source code

def get_params(self, deep=True):
    d = {
        "reg_param": self.reg_param,
        "estimator_": self.estimator_,
        "shrinkage_scheme_": self.shrinkage_scheme_,
        "max_leaf_nodes": self.estimator_.max_leaf_nodes,
    }
    if deep:
        return deepcopy(d)
    return d

def get_rules(self, feature_names=None)

Return this model's rules as a DataFrame (see imodels.get_rules).

Expand source code

def get_rules(self, feature_names=None):
    """Return this model's rules as a DataFrame (see imodels.get_rules)."""
    from imodels.util.get_rules import get_rules
    return get_rules(self, feature_names=feature_names)

def predict(self, X, *args, **kwargs)

Expand source code

def predict(self, X, *args, **kwargs):
    preds = self.estimator_.predict(X, *args, **kwargs)
    if hasattr(self.estimator_, "classes_"):
        return np.array([self.classes_[int(i)] for i in preds])
    else:
        return preds

def predict_proba(self, X, *args, **kwargs)

Expand source code

def predict_proba(self, X, *args, **kwargs):
    if hasattr(self.estimator_, "predict_proba"):
        probs = self.estimator_.predict_proba(X, *args, **kwargs)
        # the shrinkage arithmetic can leave values a hair outside [0, 1]
        # (e.g. -1e-17), which newer versions of sklearn's log_loss reject
        probs = np.clip(probs, 0, 1)
        totals = probs.sum(axis=1, keepdims=True)
        with np.errstate(invalid="ignore", divide="ignore"):
            normalized = probs / totals
        return np.where(totals > 0, normalized, 1 / probs.shape[1])
    else:
        return NotImplemented

def set_fit_request(self: HSTree, *, sample_weight: bool | str | None = '$UNCHANGED$') ‑> HSTree

Configure whether metadata should be requested to be passed to the fit method.

Note that this method is only relevant when this estimator is used as a sub-estimator within a :term:meta-estimator and metadata routing is enabled with enable_metadata_routing=True (see :func:sklearn.set_config). Please check the :ref:User Guide <metadata_routing> on how the routing mechanism works.

The options for each parameter are:

• True: metadata is requested, and passed to fit if provided. The request is ignored if metadata is not provided.

• False: metadata is not requested and the meta-estimator will not pass it to fit.

• None: metadata is not requested, and the meta-estimator will raise an error if the user provides it.

• str: metadata should be passed to the meta-estimator with this given alias instead of the original name.

The default (sklearn.utils.metadata_routing.UNCHANGED) retains the existing request. This allows you to change the request for some parameters and not others.

Added in version: 1.3

Parameters

sample_weight : str, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED

Metadata routing for sample_weight parameter in fit.

Returns

self : object

The updated object.

Expand source code

def func(*args, **kw):
    """Updates the `_metadata_request` attribute of the consumer (`instance`)
    for the parameters provided as `**kw`.

    This docstring is overwritten below.
    See REQUESTER_DOC for expected functionality.
    """
    if not _routing_enabled():
        raise RuntimeError(
            "This method is only available when metadata routing is enabled."
            " You can enable it using"
            " sklearn.set_config(enable_metadata_routing=True)."
        )

    if self.validate_keys and (set(kw) - set(self.keys)):
        raise TypeError(
            f"Unexpected args: {set(kw) - set(self.keys)} in {self.name}. "
            f"Accepted arguments are: {set(self.keys)}"
        )

    # This makes it possible to use the decorated method as an unbound method,
    # for instance when monkeypatching.
    # https://github.com/scikit-learn/scikit-learn/issues/28632
    if instance is None:
        _instance = args[0]
        args = args[1:]
    else:
        _instance = instance

    # Replicating python's behavior when positional args are given other than
    # `self`, and `self` is only allowed if this method is unbound.
    if args:
        raise TypeError(
            f"set_{self.name}_request() takes 0 positional argument but"
            f" {len(args)} were given"
        )

    requests = _instance._get_metadata_request()
    method_metadata_request = getattr(requests, self.name)

    for prop, alias in kw.items():
        if alias is not UNCHANGED:
            method_metadata_request.add_request(param=prop, alias=alias)
    _instance._metadata_request = requests

    return _instance

class HSTreeClassifier (estimator_: sklearn.base.BaseEstimator = None, reg_param: float = 1, shrinkage_scheme_: str = 'node_based', max_leaf_nodes: int = None, random_state: int = None)

Mixin class for all classifiers in scikit-learn.

This mixin defines the following functionality:

• set estimator type to "classifier" through the estimator_type tag;
• score method that default to :func:~sklearn.metrics.accuracy_score.
• enforce that fit requires y to be passed through the requires_y tag, which is done by setting the classifier type tag.

Read more in the :ref:User Guide <rolling_your_own_estimator>.

Examples

>>> import numpy as np
>>> from sklearn.base import BaseEstimator, ClassifierMixin
>>> # Mixin classes should always be on the left-hand side for a correct MRO
>>> class MyEstimator(ClassifierMixin, BaseEstimator):
...     def __init__(self, *, param=1):
...         self.param = param
...     def fit(self, X, y=None):
...         self.is_fitted_ = True
...         return self
...     def predict(self, X):
...         return np.full(shape=X.shape[0], fill_value=self.param)
>>> estimator = MyEstimator(param=1)
>>> X = np.array([[1, 2], [2, 3], [3, 4]])
>>> y = np.array([1, 0, 1])
>>> estimator.fit(X, y).predict(X)
array([1, 1, 1])
>>> estimator.score(X, y)
0.66...

Expand source code

class HSTreeClassifier(ClassifierMixin, HSTree):
    def __init__(
        self,
        estimator_: BaseEstimator = None,
        reg_param: float = 1,
        shrinkage_scheme_: str = "node_based",
        max_leaf_nodes: int = None,
        random_state: int = None,
    ):
        super().__init__(
            estimator_=estimator_,
            reg_param=reg_param,
            shrinkage_scheme_=shrinkage_scheme_,
            max_leaf_nodes=max_leaf_nodes,
            random_state=random_state,
        )

Ancestors

• sklearn.base.ClassifierMixin
• HSTree
• sklearn.base.BaseEstimator
• sklearn.utils._repr_html.base.ReprHTMLMixin
• sklearn.utils._repr_html.base._HTMLDocumentationLinkMixin
• sklearn.utils._metadata_requests._MetadataRequester

Subclasses

• HSDecisionTreeCCPClassifierCV
• HSTreeClassifierCV

Methods

def set_score_request(self: HSTreeClassifier, *, sample_weight: bool | str | None = '$UNCHANGED$') ‑> HSTreeClassifier

Configure whether metadata should be requested to be passed to the score method.

Note that this method is only relevant when this estimator is used as a sub-estimator within a :term:meta-estimator and metadata routing is enabled with enable_metadata_routing=True (see :func:sklearn.set_config). Please check the :ref:User Guide <metadata_routing> on how the routing mechanism works.

The options for each parameter are:

• True: metadata is requested, and passed to score if provided. The request is ignored if metadata is not provided.

• False: metadata is not requested and the meta-estimator will not pass it to score.

• None: metadata is not requested, and the meta-estimator will raise an error if the user provides it.

• str: metadata should be passed to the meta-estimator with this given alias instead of the original name.

The default (sklearn.utils.metadata_routing.UNCHANGED) retains the existing request. This allows you to change the request for some parameters and not others.

Added in version: 1.3

Parameters

sample_weight : str, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED

Metadata routing for sample_weight parameter in score.

Returns

self : object

The updated object.

Expand source code

def func(*args, **kw):
    """Updates the `_metadata_request` attribute of the consumer (`instance`)
    for the parameters provided as `**kw`.

    This docstring is overwritten below.
    See REQUESTER_DOC for expected functionality.
    """
    if not _routing_enabled():
        raise RuntimeError(
            "This method is only available when metadata routing is enabled."
            " You can enable it using"
            " sklearn.set_config(enable_metadata_routing=True)."
        )

    if self.validate_keys and (set(kw) - set(self.keys)):
        raise TypeError(
            f"Unexpected args: {set(kw) - set(self.keys)} in {self.name}. "
            f"Accepted arguments are: {set(self.keys)}"
        )

    # This makes it possible to use the decorated method as an unbound method,
    # for instance when monkeypatching.
    # https://github.com/scikit-learn/scikit-learn/issues/28632
    if instance is None:
        _instance = args[0]
        args = args[1:]
    else:
        _instance = instance

    # Replicating python's behavior when positional args are given other than
    # `self`, and `self` is only allowed if this method is unbound.
    if args:
        raise TypeError(
            f"set_{self.name}_request() takes 0 positional argument but"
            f" {len(args)} were given"
        )

    requests = _instance._get_metadata_request()
    method_metadata_request = getattr(requests, self.name)

    for prop, alias in kw.items():
        if alias is not UNCHANGED:
            method_metadata_request.add_request(param=prop, alias=alias)
    _instance._metadata_request = requests

    return _instance

Inherited members

• HSTree:
  • apply
  • get_params
  • get_rules
  • set_fit_request

class HSTreeClassifierCV (estimator_: sklearn.base.BaseEstimator = None, reg_param_list: List[float] = [0, 0.1, 1, 10, 50, 100, 500], shrinkage_scheme_: str = 'node_based', max_leaf_nodes: int = 20, cv: int = 3, scoring=None, *args, **kwargs)

Mixin class for all classifiers in scikit-learn.

This mixin defines the following functionality:

• set estimator type to "classifier" through the estimator_type tag;
• score method that default to :func:~sklearn.metrics.accuracy_score.
• enforce that fit requires y to be passed through the requires_y tag, which is done by setting the classifier type tag.

Read more in the :ref:User Guide <rolling_your_own_estimator>.

Examples

>>> import numpy as np
>>> from sklearn.base import BaseEstimator, ClassifierMixin
>>> # Mixin classes should always be on the left-hand side for a correct MRO
>>> class MyEstimator(ClassifierMixin, BaseEstimator):
...     def __init__(self, *, param=1):
...         self.param = param
...     def fit(self, X, y=None):
...         self.is_fitted_ = True
...         return self
...     def predict(self, X):
...         return np.full(shape=X.shape[0], fill_value=self.param)
>>> estimator = MyEstimator(param=1)
>>> X = np.array([[1, 2], [2, 3], [3, 4]])
>>> y = np.array([1, 0, 1])
>>> estimator.fit(X, y).predict(X)
array([1, 1, 1])
>>> estimator.score(X, y)
0.66...

Cross-validation is used to select the best regularization parameter for hierarchical shrinkage.

Params

estimator_ Sklearn estimator (already initialized). If no estimator_ is passed, sklearn decision tree is used

max_rules If estimator is None, then max_leaf_nodes is passed to the default decision tree

args, kwargs Note: args, kwargs are not used but left so that imodels-experiments can still pass redundant args.

Expand source code

class HSTreeClassifierCV(HSTreeClassifier):
    def __init__(
        self,
        estimator_: BaseEstimator = None,
        reg_param_list: List[float] = [0, 0.1, 1, 10, 50, 100, 500],
        shrinkage_scheme_: str = "node_based",
        max_leaf_nodes: int = 20,
        cv: int = 3,
        scoring=None,
        *args,
        **kwargs
    ):
        """Cross-validation is used to select the best regularization parameter for hierarchical shrinkage.

         Params
        ------
        estimator_
            Sklearn estimator (already initialized).
            If no estimator_ is passed, sklearn decision tree is used

        max_rules
            If estimator is None, then max_leaf_nodes is passed to the default decision tree

        args, kwargs
            Note: args, kwargs are not used but left so that imodels-experiments can still pass redundant args.
        """
        if estimator_ is None:
            estimator_ = DecisionTreeClassifier(max_leaf_nodes=max_leaf_nodes)
        super().__init__(estimator_, reg_param=None)
        # stored unmodified so that the estimator stays sklearn-cloneable
        self.reg_param_list = reg_param_list
        self.cv = cv
        self.scoring = scoring
        self.shrinkage_scheme_ = shrinkage_scheme_
        # print('estimator', self.estimator_,
        #       'checks.check_is_fitted(estimator)', checks.check_is_fitted(self.estimator_))
        # if checks.check_is_fitted(self.estimator_):
        #     raise Warning('Passed an already fitted estimator,'
        #                   'but shrinking not applied until fit method is called.')

    def get_params(self, deep=True):
        d = {
            "estimator_": self.estimator_,
            "reg_param_list": self.reg_param_list,
            "shrinkage_scheme_": self.shrinkage_scheme_,
            "max_leaf_nodes": self.estimator_.max_leaf_nodes,
            "cv": self.cv,
            "scoring": self.scoring,
        }
        if deep:
            return deepcopy(d)
        return d

    def fit(self, X, y, *args, **kwargs):
        X_arr, y_arr = _as_arrays(X, y)
        self.scores_ = [[] for _ in self.reg_param_list]
        scorer = kwargs.get("scoring", log_loss)
        kf = KFold(n_splits=self.cv)
        for train_index, test_index in kf.split(X_arr):
            X_out, y_out = X_arr[test_index, :], y_arr[test_index]
            X_in, y_in = X_arr[train_index, :], y_arr[train_index]
            base_est = deepcopy(self.estimator_)
            base_est.fit(X_in, y_in)
            for i, reg_param in enumerate(self.reg_param_list):
                # shrinkage is post-hoc, so shrink a copy of the tree fitted
                # above rather than refitting it for every reg_param
                est_hs = HSTreeClassifier(deepcopy(base_est), reg_param)
                self.scores_[i].append(
                    scorer(y_out, est_hs.predict_proba(X_out)))
        self.scores_ = [np.mean(s) for s in self.scores_]
        cv_criterion = _get_cv_criterion(scorer)
        self.reg_param = np.asarray(self.reg_param_list)[cv_criterion(self.scores_)]
        return super().fit(X=X, y=y, *args, **kwargs)

    def __repr__(self):
        attr_list = [
            "estimator_",
            "reg_param_list",
            "shrinkage_scheme_",
            "cv",
            "scoring",
        ]
        s = self.__class__.__name__
        s += "("
        for attr in attr_list:
            s += attr + "=" + repr(getattr(self, attr)) + ", "
        s = s[:-2] + ")"
        return s

Ancestors

• HSTreeClassifier
• sklearn.base.ClassifierMixin
• HSTree
• sklearn.base.BaseEstimator
• sklearn.utils._repr_html.base.ReprHTMLMixin
• sklearn.utils._repr_html.base._HTMLDocumentationLinkMixin
• sklearn.utils._metadata_requests._MetadataRequester

Methods

def fit(self, X, y, *args, **kwargs)

Expand source code

def fit(self, X, y, *args, **kwargs):
    X_arr, y_arr = _as_arrays(X, y)
    self.scores_ = [[] for _ in self.reg_param_list]
    scorer = kwargs.get("scoring", log_loss)
    kf = KFold(n_splits=self.cv)
    for train_index, test_index in kf.split(X_arr):
        X_out, y_out = X_arr[test_index, :], y_arr[test_index]
        X_in, y_in = X_arr[train_index, :], y_arr[train_index]
        base_est = deepcopy(self.estimator_)
        base_est.fit(X_in, y_in)
        for i, reg_param in enumerate(self.reg_param_list):
            # shrinkage is post-hoc, so shrink a copy of the tree fitted
            # above rather than refitting it for every reg_param
            est_hs = HSTreeClassifier(deepcopy(base_est), reg_param)
            self.scores_[i].append(
                scorer(y_out, est_hs.predict_proba(X_out)))
    self.scores_ = [np.mean(s) for s in self.scores_]
    cv_criterion = _get_cv_criterion(scorer)
    self.reg_param = np.asarray(self.reg_param_list)[cv_criterion(self.scores_)]
    return super().fit(X=X, y=y, *args, **kwargs)

def set_fit_request(self: HSTreeClassifierCV) ‑> HSTreeClassifierCV

No-op.

Calling this method has no effect.

Returns

self : object

The updated object.

Expand source code

def func(*args, **kw):
    """Updates the `_metadata_request` attribute of the consumer (`instance`)
    for the parameters provided as `**kw`.

    This docstring is overwritten below.
    See REQUESTER_DOC for expected functionality.
    """
    if not _routing_enabled():
        raise RuntimeError(
            "This method is only available when metadata routing is enabled."
            " You can enable it using"
            " sklearn.set_config(enable_metadata_routing=True)."
        )

    if self.validate_keys and (set(kw) - set(self.keys)):
        raise TypeError(
            f"Unexpected args: {set(kw) - set(self.keys)} in {self.name}. "
            f"Accepted arguments are: {set(self.keys)}"
        )

    # This makes it possible to use the decorated method as an unbound method,
    # for instance when monkeypatching.
    # https://github.com/scikit-learn/scikit-learn/issues/28632
    if instance is None:
        _instance = args[0]
        args = args[1:]
    else:
        _instance = instance

    # Replicating python's behavior when positional args are given other than
    # `self`, and `self` is only allowed if this method is unbound.
    if args:
        raise TypeError(
            f"set_{self.name}_request() takes 0 positional argument but"
            f" {len(args)} were given"
        )

    requests = _instance._get_metadata_request()
    method_metadata_request = getattr(requests, self.name)

    for prop, alias in kw.items():
        if alias is not UNCHANGED:
            method_metadata_request.add_request(param=prop, alias=alias)
    _instance._metadata_request = requests

    return _instance

Inherited members

• HSTreeClassifier:
  • apply
  • get_params
  • get_rules
  • set_score_request

class HSTreeRegressor (estimator_: sklearn.base.BaseEstimator = None, reg_param: float = 1, shrinkage_scheme_: str = 'node_based', max_leaf_nodes: int = None, random_state: int = None)

Mixin class for all regression estimators in scikit-learn.

This mixin defines the following functionality:

• set estimator type to "regressor" through the estimator_type tag;
• score method that default to :func:~sklearn.metrics.r2_score.
• enforce that fit requires y to be passed through the requires_y tag, which is done by setting the regressor type tag.

Read more in the :ref:User Guide <rolling_your_own_estimator>.

Examples

>>> import numpy as np
>>> from sklearn.base import BaseEstimator, RegressorMixin
>>> # Mixin classes should always be on the left-hand side for a correct MRO
>>> class MyEstimator(RegressorMixin, BaseEstimator):
...     def __init__(self, *, param=1):
...         self.param = param
...     def fit(self, X, y=None):
...         self.is_fitted_ = True
...         return self
...     def predict(self, X):
...         return np.full(shape=X.shape[0], fill_value=self.param)
>>> estimator = MyEstimator(param=0)
>>> X = np.array([[1, 2], [2, 3], [3, 4]])
>>> y = np.array([-1, 0, 1])
>>> estimator.fit(X, y).predict(X)
array([0, 0, 0])
>>> estimator.score(X, y)
0.0

Expand source code

class HSTreeRegressor(RegressorMixin, HSTree):
    def __init__(
        self,
        estimator_: BaseEstimator = None,
        reg_param: float = 1,
        shrinkage_scheme_: str = "node_based",
        max_leaf_nodes: int = None,
        random_state: int = None,
    ):
        if estimator_ is None:
            estimator_ = DecisionTreeRegressor(max_leaf_nodes=20)
        if estimator_ is None:
            estimator_ = DecisionTreeClassifier(max_leaf_nodes=20)
        super().__init__(
            estimator_=estimator_,
            reg_param=reg_param,
            shrinkage_scheme_=shrinkage_scheme_,
            max_leaf_nodes=max_leaf_nodes,
            random_state=random_state,
        )

Ancestors

• sklearn.base.RegressorMixin
• HSTree
• sklearn.base.BaseEstimator
• sklearn.utils._repr_html.base.ReprHTMLMixin
• sklearn.utils._repr_html.base._HTMLDocumentationLinkMixin
• sklearn.utils._metadata_requests._MetadataRequester

Subclasses

• HSDecisionTreeCCPRegressorCV
• HSTreeRegressorCV

Methods

def set_score_request(self: HSTreeRegressor, *, sample_weight: bool | str | None = '$UNCHANGED$') ‑> HSTreeRegressor

Configure whether metadata should be requested to be passed to the score method.

Note that this method is only relevant when this estimator is used as a sub-estimator within a :term:meta-estimator and metadata routing is enabled with enable_metadata_routing=True (see :func:sklearn.set_config). Please check the :ref:User Guide <metadata_routing> on how the routing mechanism works.

The options for each parameter are:

• True: metadata is requested, and passed to score if provided. The request is ignored if metadata is not provided.

• False: metadata is not requested and the meta-estimator will not pass it to score.

• None: metadata is not requested, and the meta-estimator will raise an error if the user provides it.

• str: metadata should be passed to the meta-estimator with this given alias instead of the original name.

The default (sklearn.utils.metadata_routing.UNCHANGED) retains the existing request. This allows you to change the request for some parameters and not others.

Added in version: 1.3

Parameters

sample_weight : str, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED

Metadata routing for sample_weight parameter in score.

Returns

self : object

The updated object.

Expand source code

def func(*args, **kw):
    """Updates the `_metadata_request` attribute of the consumer (`instance`)
    for the parameters provided as `**kw`.

    This docstring is overwritten below.
    See REQUESTER_DOC for expected functionality.
    """
    if not _routing_enabled():
        raise RuntimeError(
            "This method is only available when metadata routing is enabled."
            " You can enable it using"
            " sklearn.set_config(enable_metadata_routing=True)."
        )

    if self.validate_keys and (set(kw) - set(self.keys)):
        raise TypeError(
            f"Unexpected args: {set(kw) - set(self.keys)} in {self.name}. "
            f"Accepted arguments are: {set(self.keys)}"
        )

    # This makes it possible to use the decorated method as an unbound method,
    # for instance when monkeypatching.
    # https://github.com/scikit-learn/scikit-learn/issues/28632
    if instance is None:
        _instance = args[0]
        args = args[1:]
    else:
        _instance = instance

    # Replicating python's behavior when positional args are given other than
    # `self`, and `self` is only allowed if this method is unbound.
    if args:
        raise TypeError(
            f"set_{self.name}_request() takes 0 positional argument but"
            f" {len(args)} were given"
        )

    requests = _instance._get_metadata_request()
    method_metadata_request = getattr(requests, self.name)

    for prop, alias in kw.items():
        if alias is not UNCHANGED:
            method_metadata_request.add_request(param=prop, alias=alias)
    _instance._metadata_request = requests

    return _instance

Inherited members

• HSTree:
  • apply
  • get_params
  • get_rules
  • set_fit_request

class HSTreeRegressorCV (estimator_: sklearn.base.BaseEstimator = None, reg_param_list: List[float] = [0, 0.1, 1, 10, 50, 100, 500], shrinkage_scheme_: str = 'node_based', max_leaf_nodes: int = 20, cv: int = 3, scoring=None, *args, **kwargs)

Mixin class for all regression estimators in scikit-learn.

This mixin defines the following functionality:

• set estimator type to "regressor" through the estimator_type tag;
• score method that default to :func:~sklearn.metrics.r2_score.
• enforce that fit requires y to be passed through the requires_y tag, which is done by setting the regressor type tag.

Read more in the :ref:User Guide <rolling_your_own_estimator>.

Examples

>>> import numpy as np
>>> from sklearn.base import BaseEstimator, RegressorMixin
>>> # Mixin classes should always be on the left-hand side for a correct MRO
>>> class MyEstimator(RegressorMixin, BaseEstimator):
...     def __init__(self, *, param=1):
...         self.param = param
...     def fit(self, X, y=None):
...         self.is_fitted_ = True
...         return self
...     def predict(self, X):
...         return np.full(shape=X.shape[0], fill_value=self.param)
>>> estimator = MyEstimator(param=0)
>>> X = np.array([[1, 2], [2, 3], [3, 4]])
>>> y = np.array([-1, 0, 1])
>>> estimator.fit(X, y).predict(X)
array([0, 0, 0])
>>> estimator.score(X, y)
0.0

Cross-validation is used to select the best regularization parameter for hierarchical shrinkage.

Params

estimator_ Sklearn estimator (already initialized). If no estimator_ is passed, sklearn decision tree is used

max_rules If estimator is None, then max_leaf_nodes is passed to the default decision tree

args, kwargs Note: args, kwargs are not used but left so that imodels-experiments can still pass redundant args.

Expand source code

class HSTreeRegressorCV(HSTreeRegressor):
    def __init__(
        self,
        estimator_: BaseEstimator = None,
        reg_param_list: List[float] = [0, 0.1, 1, 10, 50, 100, 500],
        shrinkage_scheme_: str = "node_based",
        max_leaf_nodes: int = 20,
        cv: int = 3,
        scoring=None,
        *args,
        **kwargs
    ):
        """Cross-validation is used to select the best regularization parameter for hierarchical shrinkage.

         Params
        ------
        estimator_
            Sklearn estimator (already initialized).
            If no estimator_ is passed, sklearn decision tree is used

        max_rules
            If estimator is None, then max_leaf_nodes is passed to the default decision tree

        args, kwargs
            Note: args, kwargs are not used but left so that imodels-experiments can still pass redundant args.
        """
        if estimator_ is None:
            estimator_ = DecisionTreeRegressor(max_leaf_nodes=max_leaf_nodes)
        super().__init__(estimator_, reg_param=None)
        # stored unmodified so that the estimator stays sklearn-cloneable
        self.reg_param_list = reg_param_list
        self.cv = cv
        self.scoring = scoring
        self.shrinkage_scheme_ = shrinkage_scheme_
        # print('estimator', self.estimator_,
        #       'checks.check_is_fitted(estimator)', checks.check_is_fitted(self.estimator_))
        # if checks.check_is_fitted(self.estimator_):
        #     raise Warning('Passed an already fitted estimator,'
        #                   'but shrinking not applied until fit method is called.')

    def get_params(self, deep=True):
        d = {
            "estimator_": self.estimator_,
            "reg_param_list": self.reg_param_list,
            "shrinkage_scheme_": self.shrinkage_scheme_,
            "max_leaf_nodes": self.estimator_.max_leaf_nodes,
            "cv": self.cv,
            "scoring": self.scoring,
        }
        if deep:
            return deepcopy(d)
        return d

    def fit(self, X, y, *args, **kwargs):
        X_arr, y_arr = _as_arrays(X, y)
        self.scores_ = [[] for _ in self.reg_param_list]
        kf = KFold(n_splits=self.cv)
        scorer = kwargs.get("scoring", mean_squared_error)
        for train_index, test_index in kf.split(X_arr):
            X_out, y_out = X_arr[test_index, :], y_arr[test_index]
            X_in, y_in = X_arr[train_index, :], y_arr[train_index]
            base_est = deepcopy(self.estimator_)
            base_est.fit(X_in, y_in)
            for i, reg_param in enumerate(self.reg_param_list):
                # shrinkage is post-hoc, so shrink a copy of the tree fitted
                # above rather than refitting it for every reg_param
                est_hs = HSTreeRegressor(deepcopy(base_est), reg_param)
                self.scores_[i].append(scorer(est_hs.predict(X_out), y_out))
        self.scores_ = [np.mean(s) for s in self.scores_]
        cv_criterion = _get_cv_criterion(scorer)
        self.reg_param = np.asarray(self.reg_param_list)[cv_criterion(self.scores_)]
        return super().fit(X=X, y=y, *args, **kwargs)

    def __repr__(self):
        attr_list = [
            "estimator_",
            "reg_param_list",
            "shrinkage_scheme_",
            "cv",
            "scoring",
        ]
        s = self.__class__.__name__
        s += "("
        for attr in attr_list:
            s += attr + "=" + repr(getattr(self, attr)) + ", "
        s = s[:-2] + ")"
        return s

Ancestors

• HSTreeRegressor
• sklearn.base.RegressorMixin
• HSTree
• sklearn.base.BaseEstimator
• sklearn.utils._repr_html.base.ReprHTMLMixin
• sklearn.utils._repr_html.base._HTMLDocumentationLinkMixin
• sklearn.utils._metadata_requests._MetadataRequester

Methods

def fit(self, X, y, *args, **kwargs)

Expand source code

def fit(self, X, y, *args, **kwargs):
    X_arr, y_arr = _as_arrays(X, y)
    self.scores_ = [[] for _ in self.reg_param_list]
    kf = KFold(n_splits=self.cv)
    scorer = kwargs.get("scoring", mean_squared_error)
    for train_index, test_index in kf.split(X_arr):
        X_out, y_out = X_arr[test_index, :], y_arr[test_index]
        X_in, y_in = X_arr[train_index, :], y_arr[train_index]
        base_est = deepcopy(self.estimator_)
        base_est.fit(X_in, y_in)
        for i, reg_param in enumerate(self.reg_param_list):
            # shrinkage is post-hoc, so shrink a copy of the tree fitted
            # above rather than refitting it for every reg_param
            est_hs = HSTreeRegressor(deepcopy(base_est), reg_param)
            self.scores_[i].append(scorer(est_hs.predict(X_out), y_out))
    self.scores_ = [np.mean(s) for s in self.scores_]
    cv_criterion = _get_cv_criterion(scorer)
    self.reg_param = np.asarray(self.reg_param_list)[cv_criterion(self.scores_)]
    return super().fit(X=X, y=y, *args, **kwargs)

def set_fit_request(self: HSTreeRegressorCV) ‑> HSTreeRegressorCV

No-op.

Calling this method has no effect.

Returns

self : object

The updated object.

Expand source code

def func(*args, **kw):
    """Updates the `_metadata_request` attribute of the consumer (`instance`)
    for the parameters provided as `**kw`.

    This docstring is overwritten below.
    See REQUESTER_DOC for expected functionality.
    """
    if not _routing_enabled():
        raise RuntimeError(
            "This method is only available when metadata routing is enabled."
            " You can enable it using"
            " sklearn.set_config(enable_metadata_routing=True)."
        )

    if self.validate_keys and (set(kw) - set(self.keys)):
        raise TypeError(
            f"Unexpected args: {set(kw) - set(self.keys)} in {self.name}. "
            f"Accepted arguments are: {set(self.keys)}"
        )

    # This makes it possible to use the decorated method as an unbound method,
    # for instance when monkeypatching.
    # https://github.com/scikit-learn/scikit-learn/issues/28632
    if instance is None:
        _instance = args[0]
        args = args[1:]
    else:
        _instance = instance

    # Replicating python's behavior when positional args are given other than
    # `self`, and `self` is only allowed if this method is unbound.
    if args:
        raise TypeError(
            f"set_{self.name}_request() takes 0 positional argument but"
            f" {len(args)} were given"
        )

    requests = _instance._get_metadata_request()
    method_metadata_request = getattr(requests, self.name)

    for prop, alias in kw.items():
        if alias is not UNCHANGED:
            method_metadata_request.add_request(param=prop, alias=alias)
    _instance._metadata_request = requests

    return _instance

Inherited members

• HSTreeRegressor:
  • apply
  • get_params
  • get_rules
  • set_score_request