smile.classification

Class AdaBoost

java.lang.Object

smile.classification.AdaBoost

All Implemented Interfaces:

Classifier<double[]>

public class AdaBoost
extends Object
implements Classifier<double[]>

AdaBoost (Adaptive Boosting) classifier with decision trees. In principle, AdaBoost is a meta-algorithm, and can be used in conjunction with many other learning algorithms to improve their performance. In practice, AdaBoost with decision trees is probably the most popular combination. AdaBoost is adaptive in the sense that subsequent classifiers built are tweaked in favor of those instances misclassified by previous classifiers. AdaBoost is sensitive to noisy data and outliers. However in some problems it can be less susceptible to the over-fitting problem than most learning algorithms.

AdaBoost calls a weak classifier repeatedly in a series of rounds from total T classifiers. For each call a distribution of weights is updated that indicates the importance of examples in the data set for the classification. On each round, the weights of each incorrectly classified example are increased (or alternatively, the weights of each correctly classified example are decreased), so that the new classifier focuses more on those examples.

The basic AdaBoost algorithm is only for binary classification problem. For multi-class classification, a common approach is reducing the multi-class classification problem to multiple two-class problems. This implementation is a multi-class AdaBoost without such reductions.

References

1. Yoav Freund, Robert E. Schapire. A Decision-Theoretic Generalization of on-Line Learning and an Application to Boosting, 1995.
2. Ji Zhu, Hui Zhou, Saharon Rosset and Trevor Hastie. Multi-class Adaboost, 2009.

Author:

Haifeng Li

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	AdaBoost.Trainer Trainer for AdaBoost classifiers.

Constructor Summary

Constructors

Constructor and Description
AdaBoost(Attribute[] attributes, double[][] x, int[] y, int T) Constructor.
AdaBoost(Attribute[] attributes, double[][] x, int[] y, int T, int J) Constructor.
AdaBoost(double[][] x, int[] y, int T) Constructor.
AdaBoost(double[][] x, int[] y, int T, int J) Constructor.

Method Summary

Methods

Modifier and Type	Method and Description
double[]	importance() Returns the variable importance.
int	predict(double[] x) Predicts the class label of an instance.
int	predict(double[] x, double[] posteriori) Predicts the class label of an instance and also calculate a posteriori probabilities.
int	size() Returns the number of trees in the model.
double[]	test(double[][] x, int[] y) Test the model on a validation dataset.
double[][]	test(double[][] x, int[] y, ClassificationMeasure[] measures) Test the model on a validation dataset.
void	trim(int T) Trims the tree model set to a smaller size in case of over-fitting.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

AdaBoost

public AdaBoost(double[][] x,
        int[] y,
        int T)

Constructor. Learns AdaBoost with decision stumps.

Parameters:

x - the training instances.

y - the response variable.

T - the number of trees.

AdaBoost

public AdaBoost(double[][] x,
        int[] y,
        int T,
        int J)

Constructor. Learns AdaBoost with decision trees.

Parameters:

x - the training instances.

y - the response variable.

T - the number of trees.

J - the maximum number of leaf nodes in the trees.

AdaBoost

public AdaBoost(Attribute[] attributes,
        double[][] x,
        int[] y,
        int T)

Constructor. Learns AdaBoost with decision stumps.

Parameters:

attributes - the attribute properties.

x - the training instances.

y - the response variable.

T - the number of trees.

AdaBoost

public AdaBoost(Attribute[] attributes,
        double[][] x,
        int[] y,
        int T,
        int J)

Constructor.

Parameters:

attributes - the attribute properties.

x - the training instances.

y - the response variable.

T - the number of trees.

J - the maximum number of leaf nodes in the trees.

Method Detail

importance

public double[] importance()

Returns the variable importance. Every time a split of a node is made on variable the (GINI, information gain, etc.) impurity criterion for the two descendent nodes is less than the parent node. Adding up the decreases for each individual variable over all trees in the forest gives a simple measure of variable importance.

Returns:

the variable importance

size

public int size()

Returns the number of trees in the model.

Returns:

the number of trees in the model

trim

public void trim(int T)

Trims the tree model set to a smaller size in case of over-fitting. Or if extra decision trees in the model don't improve the performance, we may remove them to reduce the model size and also improve the speed of prediction.

Parameters:

T - the new (smaller) size of tree model set.

predict

public int predict(double[] x)

Description copied from interface: Classifier

Predicts the class label of an instance.

Specified by:

predict in interface Classifier<double[]>

Parameters:

x - the instance to be classified.

Returns:

the predicted class label

predict

public int predict(double[] x,
          double[] posteriori)

Predicts the class label of an instance and also calculate a posteriori probabilities. Not supported.

Specified by:

predict in interface Classifier<double[]>

Parameters:

x - the instance to be classified.

posteriori - the array to store a posteriori probabilities on output.

Returns:

the predicted class label

test

public double[] test(double[][] x,
            int[] y)

Test the model on a validation dataset.

Parameters:

x - the test data set.

y - the test data response values.

Returns:

accuracies with first 1, 2, ..., decision trees.

test

public double[][] test(double[][] x,
              int[] y,
              ClassificationMeasure[] measures)

Test the model on a validation dataset.

Parameters:

x - the test data set.

y - the test data labels.

measures - the performance measures of classification.

Returns:

performance measures with first 1, 2, ..., decision trees.