org.deeplearning4j.eval

Class EvaluationCalibration

java.lang.Object

org.deeplearning4j.eval.BaseEvaluation<EvaluationCalibration>

org.deeplearning4j.eval.EvaluationCalibration

All Implemented Interfaces:

Serializable, IEvaluation<EvaluationCalibration>

public class EvaluationCalibration
extends BaseEvaluation<EvaluationCalibration>

EvaluationCalibration is an evaluation class designed to analyze the calibration of a classifier.
It provides a number of tools for this purpose: - Counts of the number of labels and predictions for each class
- Reliability diagram (or reliability curve)
- Residual plot (histogram)
- Histograms of probabilities, including probabilities for each class separately

References:
- Reliability diagram: see for example Niculescu-Mizil and Caruana 2005, Predicting Good Probabilities With Supervised Learning
- Residual plot: see Wallace and Dahabreh 2012, Class Probability Estimates are Unreliable for Imbalanced Data (and How to Fix Them)

Author:

Alex Black

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
static int	DEFAULT_HISTOGRAM_NUM_BINS
static int	DEFAULT_RELIABILITY_DIAG_NUM_BINS

Constructor Summary

Constructors

Constructor and Description
EvaluationCalibration() Create an EvaluationCalibration instance with the default number of bins
EvaluationCalibration(int reliabilityDiagNumBins, int histogramNumBins) Create an EvaluationCalibration instance with the specified number of bins
EvaluationCalibration(int reliabilityDiagNumBins, int histogramNumBins, boolean excludeEmptyBins) Create an EvaluationCalibration instance with the specified number of bins

Method Summary

Modifier and Type	Method and Description
void	eval(org.nd4j.linalg.api.ndarray.INDArray labels, org.nd4j.linalg.api.ndarray.INDArray networkPredictions)
void	eval(org.nd4j.linalg.api.ndarray.INDArray labels, org.nd4j.linalg.api.ndarray.INDArray networkPredictions, org.nd4j.linalg.api.ndarray.INDArray maskArray)
int[]	getLabelCountsEachClass()
int[]	getPredictionCountsEachClass()
Histogram	getProbabilityHistogram(int labelClassIdx) Return a probability histogram of the specified label class index.
Histogram	getProbabilityHistogramAllClasses() Return a probability histogram for all predictions/classes.
ReliabilityDiagram	getReliabilityDiagram(int classIdx) Get the reliability diagram for the specified class
Histogram	getResidualPlot(int labelClassIdx) Get the residual plot, only for examples of the specified class..
Histogram	getResidualPlotAllClasses() Get the residual plot for all classes combined.
void	merge(EvaluationCalibration other)
int	numClasses()
void	reset()
String	stats()

Methods inherited from class org.deeplearning4j.eval.BaseEvaluation

equals, eval, evalTimeSeries, evalTimeSeries, fromJson, fromYaml, toJson, toString, toYaml

Methods inherited from class java.lang.Object

clone, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

DEFAULT_RELIABILITY_DIAG_NUM_BINS

public static final int DEFAULT_RELIABILITY_DIAG_NUM_BINS

See Also:

Constant Field Values

DEFAULT_HISTOGRAM_NUM_BINS

public static final int DEFAULT_HISTOGRAM_NUM_BINS

See Also:

Constant Field Values

Constructor Detail

EvaluationCalibration

public EvaluationCalibration()

Create an EvaluationCalibration instance with the default number of bins

EvaluationCalibration

public EvaluationCalibration(int reliabilityDiagNumBins,
                             int histogramNumBins)

Create an EvaluationCalibration instance with the specified number of bins

Parameters:

reliabilityDiagNumBins - Number of bins for the reliability diagram (usually 10)

histogramNumBins - Number of bins for the histograms

EvaluationCalibration

public EvaluationCalibration(int reliabilityDiagNumBins,
                             int histogramNumBins,
                             boolean excludeEmptyBins)

Create an EvaluationCalibration instance with the specified number of bins

Parameters:

reliabilityDiagNumBins - Number of bins for the reliability diagram (usually 10)

histogramNumBins - Number of bins for the histograms

excludeEmptyBins - For the reliability diagram, whether empty bins should be excluded

Method Detail

eval

public void eval(org.nd4j.linalg.api.ndarray.INDArray labels,
                 org.nd4j.linalg.api.ndarray.INDArray networkPredictions,
                 org.nd4j.linalg.api.ndarray.INDArray maskArray)

Specified by:

eval in interface IEvaluation<EvaluationCalibration>

Overrides:

eval in class BaseEvaluation<EvaluationCalibration>

eval

public void eval(org.nd4j.linalg.api.ndarray.INDArray labels,
                 org.nd4j.linalg.api.ndarray.INDArray networkPredictions)

merge

public void merge(EvaluationCalibration other)

reset

public void reset()

stats

public String stats()

Returns:

numClasses

public int numClasses()

getReliabilityDiagram

public ReliabilityDiagram getReliabilityDiagram(int classIdx)

Get the reliability diagram for the specified class

Parameters:

classIdx - Index of the class to get the reliability diagram for

getLabelCountsEachClass

public int[] getLabelCountsEachClass()

Returns:

The number of observed labels for each class. For N classes, be returned array is of length N, with out[i] being the number of labels of class i

getPredictionCountsEachClass

public int[] getPredictionCountsEachClass()

Returns:

The number of network predictions for each class. For N classes, be returned array is of length N, with out[i] being the number of predicted values (max probability) for class i

getResidualPlotAllClasses

public Histogram getResidualPlotAllClasses()

Get the residual plot for all classes combined. The residual plot is defined as a histogram of
|label_i - prob(class_i | input)| for all classes i and examples.
In general, small residuals indicate a superior classifier to large residuals.

Returns:

Residual plot (histogram) - all predictions/classes

getResidualPlot

public Histogram getResidualPlot(int labelClassIdx)

Get the residual plot, only for examples of the specified class.. The residual plot is defined as a histogram of
|label_i - prob(class_i | input)| for all and examples; for this particular method, only predictions where i == labelClassIdx are included.
In general, small residuals indicate a superior classifier to large residuals.

Parameters:

labelClassIdx - Index of the class to get the residual plot for

Returns:

Residual plot (histogram) - all predictions/classes

getProbabilityHistogramAllClasses

public Histogram getProbabilityHistogramAllClasses()

Return a probability histogram for all predictions/classes.

Returns:

Probability histogram

getProbabilityHistogram

public Histogram getProbabilityHistogram(int labelClassIdx)

Return a probability histogram of the specified label class index. That is, for label class index i, a histogram of P(class_i | input) is returned, only for those examples that are labelled as class i.

Parameters:

labelClassIdx - Index of the label class to get the histogram for

Returns:

Probability histogram