org.deeplearning4j.gradientcheck

Class GradientCheckUtil

java.lang.Object

org.deeplearning4j.gradientcheck.GradientCheckUtil

public class GradientCheckUtil
extends Object

A utility for numerically checking gradients.
Basic idea: compare calculated gradients with those calculated numerically, to check implementation of backpropagation gradient calculation.
See:
- http://cs231n.github.io/neural-networks-3/#gradcheck
- http://ufldl.stanford.edu/wiki/index.php/Gradient_checking_and_advanced_optimization
- https://code.google.com/p/cuda-convnet/wiki/CheckingGradients
Is C is cost function, then dC/dW ~= (C(w+epsilon)-C(w-epsilon)) / (2*epsilon).
Method checks gradient calculation for every parameter separately by doing 2 forward pass calculations for each parameter, so can be very time consuming for large networks.

Author:

Alex Black

Method Summary

Modifier and Type	Method and Description
static boolean	checkGradients(ComputationGraph graph, double epsilon, double maxRelError, double minAbsoluteError, boolean print, boolean exitOnFirstError, INDArray[] inputs, INDArray[] labels) Check backprop gradients for a ComputationGraph
static boolean	checkGradients(ComputationGraph graph, double epsilon, double maxRelError, double minAbsoluteError, boolean print, boolean exitOnFirstError, INDArray[] inputs, INDArray[] labels, INDArray[] fMask, INDArray[] lMask)
static boolean	checkGradients(ComputationGraph graph, double epsilon, double maxRelError, double minAbsoluteError, boolean print, boolean exitOnFirstError, INDArray[] inputs, INDArray[] labels, INDArray[] fMask, INDArray[] lMask, Set<String> excludeParams)
static boolean	checkGradients(ComputationGraph graph, double epsilon, double maxRelError, double minAbsoluteError, boolean print, boolean exitOnFirstError, INDArray[] inputs, INDArray[] labels, INDArray[] fMask, INDArray[] lMask, Set<String> excludeParams, Consumer<ComputationGraph> callEachIter)
static boolean	checkGradients(ComputationGraph graph, double epsilon, double maxRelError, double minAbsoluteError, boolean print, boolean exitOnFirstError, INDArray[] inputs, INDArray[] labels, INDArray[] fMask, INDArray[] lMask, Set<String> excludeParams, Integer rngSeedResetEachIter)
static boolean	checkGradients(MultiLayerNetwork mln, double epsilon, double maxRelError, double minAbsoluteError, boolean print, boolean exitOnFirstError, INDArray input, INDArray labels) Check backprop gradients for a MultiLayerNetwork.
static boolean	checkGradients(MultiLayerNetwork mln, double epsilon, double maxRelError, double minAbsoluteError, boolean print, boolean exitOnFirstError, INDArray input, INDArray labels, INDArray inputMask, INDArray labelMask)
static boolean	checkGradients(MultiLayerNetwork mln, double epsilon, double maxRelError, double minAbsoluteError, boolean print, boolean exitOnFirstError, INDArray input, INDArray labels, INDArray inputMask, INDArray labelMask, boolean subset, int maxPerParam)
static boolean	checkGradients(MultiLayerNetwork mln, double epsilon, double maxRelError, double minAbsoluteError, boolean print, boolean exitOnFirstError, INDArray input, INDArray labels, INDArray inputMask, INDArray labelMask, boolean subset, int maxPerParam, Set<String> excludeParams)
static boolean	checkGradients(MultiLayerNetwork mln, double epsilon, double maxRelError, double minAbsoluteError, boolean print, boolean exitOnFirstError, INDArray input, INDArray labels, INDArray inputMask, INDArray labelMask, boolean subset, int maxPerParam, Set<String> excludeParams, Consumer<MultiLayerNetwork> callEachIter)
static boolean	checkGradients(MultiLayerNetwork mln, double epsilon, double maxRelError, double minAbsoluteError, boolean print, boolean exitOnFirstError, INDArray input, INDArray labels, INDArray inputMask, INDArray labelMask, boolean subset, int maxPerParam, Set<String> excludeParams, Integer rngSeedResetEachIter)
static boolean	checkGradients(MultiLayerNetwork mln, double epsilon, double maxRelError, double minAbsoluteError, boolean print, boolean exitOnFirstError, INDArray input, INDArray labels, Set<String> excludeParams)
static boolean	checkGradientsPretrainLayer(Layer layer, double epsilon, double maxRelError, double minAbsoluteError, boolean print, boolean exitOnFirstError, INDArray input, int rngSeed) Check backprop gradients for a pretrain layer NOTE: gradient checking pretrain layers can be difficult...

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

checkGradients

public static boolean checkGradients(MultiLayerNetwork mln,
                                     double epsilon,
                                     double maxRelError,
                                     double minAbsoluteError,
                                     boolean print,
                                     boolean exitOnFirstError,
                                     INDArray input,
                                     INDArray labels)

Check backprop gradients for a MultiLayerNetwork.

Parameters:

mln - MultiLayerNetwork to test. This must be initialized.

epsilon - Usually on the order/ of 1e-4 or so.

maxRelError - Maximum relative error. Usually < 1e-5 or so, though maybe more for deep networks or those with nonlinear activation

minAbsoluteError - Minimum absolute error to cause a failure. Numerical gradients can be non-zero due to precision issues. For example, 0.0 vs. 1e-18: relative error is 1.0, but not really a failure

print - Whether to print full pass/failure details for each parameter gradient

exitOnFirstError - If true: return upon first failure. If false: continue checking even if one parameter gradient has failed. Typically use false for debugging, true for unit tests.

input - Input array to use for forward pass. May be mini-batch data.

labels - Labels/targets to use to calculate backprop gradient. May be mini-batch data.

Returns:

true if gradients are passed, false otherwise.

checkGradients

public static boolean checkGradients(MultiLayerNetwork mln,
                                     double epsilon,
                                     double maxRelError,
                                     double minAbsoluteError,
                                     boolean print,
                                     boolean exitOnFirstError,
                                     INDArray input,
                                     INDArray labels,
                                     Set<String> excludeParams)

checkGradients

public static boolean checkGradients(MultiLayerNetwork mln,
                                     double epsilon,
                                     double maxRelError,
                                     double minAbsoluteError,
                                     boolean print,
                                     boolean exitOnFirstError,
                                     INDArray input,
                                     INDArray labels,
                                     INDArray inputMask,
                                     INDArray labelMask)

checkGradients

public static boolean checkGradients(MultiLayerNetwork mln,
                                     double epsilon,
                                     double maxRelError,
                                     double minAbsoluteError,
                                     boolean print,
                                     boolean exitOnFirstError,
                                     INDArray input,
                                     INDArray labels,
                                     INDArray inputMask,
                                     INDArray labelMask,
                                     boolean subset,
                                     int maxPerParam)

checkGradients

public static boolean checkGradients(MultiLayerNetwork mln,
                                     double epsilon,
                                     double maxRelError,
                                     double minAbsoluteError,
                                     boolean print,
                                     boolean exitOnFirstError,
                                     INDArray input,
                                     INDArray labels,
                                     INDArray inputMask,
                                     INDArray labelMask,
                                     boolean subset,
                                     int maxPerParam,
                                     Set<String> excludeParams)

checkGradients

public static boolean checkGradients(MultiLayerNetwork mln,
                                     double epsilon,
                                     double maxRelError,
                                     double minAbsoluteError,
                                     boolean print,
                                     boolean exitOnFirstError,
                                     INDArray input,
                                     INDArray labels,
                                     INDArray inputMask,
                                     INDArray labelMask,
                                     boolean subset,
                                     int maxPerParam,
                                     Set<String> excludeParams,
                                     Integer rngSeedResetEachIter)

checkGradients

public static boolean checkGradients(MultiLayerNetwork mln,
                                     double epsilon,
                                     double maxRelError,
                                     double minAbsoluteError,
                                     boolean print,
                                     boolean exitOnFirstError,
                                     INDArray input,
                                     INDArray labels,
                                     INDArray inputMask,
                                     INDArray labelMask,
                                     boolean subset,
                                     int maxPerParam,
                                     Set<String> excludeParams,
                                     Consumer<MultiLayerNetwork> callEachIter)

checkGradients

public static boolean checkGradients(ComputationGraph graph,
                                     double epsilon,
                                     double maxRelError,
                                     double minAbsoluteError,
                                     boolean print,
                                     boolean exitOnFirstError,
                                     INDArray[] inputs,
                                     INDArray[] labels)

Check backprop gradients for a ComputationGraph

Parameters:

graph - ComputationGraph to test. This must be initialized.

epsilon - Usually on the order of 1e-4 or so.

maxRelError - Maximum relative error. Usually < 0.01, though maybe more for deep networks

minAbsoluteError - Minimum absolute error to cause a failure. Numerical gradients can be non-zero due to precision issues. For example, 0.0 vs. 1e-18: relative error is 1.0, but not really a failure

print - Whether to print full pass/failure details for each parameter gradient

exitOnFirstError - If true: return upon first failure. If false: continue checking even if one parameter gradient has failed. Typically use false for debugging, true for unit tests.

inputs - Input arrays to use for forward pass. May be mini-batch data.

labels - Labels/targets (output) arrays to use to calculate backprop gradient. May be mini-batch data.

Returns:

true if gradients are passed, false otherwise.

checkGradients

public static boolean checkGradients(ComputationGraph graph,
                                     double epsilon,
                                     double maxRelError,
                                     double minAbsoluteError,
                                     boolean print,
                                     boolean exitOnFirstError,
                                     INDArray[] inputs,
                                     INDArray[] labels,
                                     INDArray[] fMask,
                                     INDArray[] lMask)

checkGradients

public static boolean checkGradients(ComputationGraph graph,
                                     double epsilon,
                                     double maxRelError,
                                     double minAbsoluteError,
                                     boolean print,
                                     boolean exitOnFirstError,
                                     INDArray[] inputs,
                                     INDArray[] labels,
                                     INDArray[] fMask,
                                     INDArray[] lMask,
                                     Set<String> excludeParams)

checkGradients

public static boolean checkGradients(ComputationGraph graph,
                                     double epsilon,
                                     double maxRelError,
                                     double minAbsoluteError,
                                     boolean print,
                                     boolean exitOnFirstError,
                                     INDArray[] inputs,
                                     INDArray[] labels,
                                     INDArray[] fMask,
                                     INDArray[] lMask,
                                     Set<String> excludeParams,
                                     Integer rngSeedResetEachIter)

checkGradients

public static boolean checkGradients(ComputationGraph graph,
                                     double epsilon,
                                     double maxRelError,
                                     double minAbsoluteError,
                                     boolean print,
                                     boolean exitOnFirstError,
                                     INDArray[] inputs,
                                     INDArray[] labels,
                                     INDArray[] fMask,
                                     INDArray[] lMask,
                                     Set<String> excludeParams,
                                     Consumer<ComputationGraph> callEachIter)

checkGradientsPretrainLayer

public static boolean checkGradientsPretrainLayer(Layer layer,
                                                  double epsilon,
                                                  double maxRelError,
                                                  double minAbsoluteError,
                                                  boolean print,
                                                  boolean exitOnFirstError,
                                                  INDArray input,
                                                  int rngSeed)

Check backprop gradients for a pretrain layer NOTE: gradient checking pretrain layers can be difficult...