org.deeplearning4j.nn.conf.layers

Class BatchNormalization.Builder

java.lang.Object

org.deeplearning4j.nn.conf.layers.Layer.Builder<T>

org.deeplearning4j.nn.conf.layers.BaseLayer.Builder<T>

org.deeplearning4j.nn.conf.layers.FeedForwardLayer.Builder<BatchNormalization.Builder>

org.deeplearning4j.nn.conf.layers.BatchNormalization.Builder

Enclosing class:

BatchNormalization

public static class BatchNormalization.Builder
extends FeedForwardLayer.Builder<BatchNormalization.Builder>

Field Summary

Fields

Modifier and Type	Field and Description
protected double	beta Used only when 'true' is passed to lockGammaBeta(boolean).
protected List<LayerConstraint>	betaConstraints Set constraints to be applied to the beta parameter of this batch normalisation layer.
protected boolean	cudnnAllowFallback When using CuDNN and an error is encountered, should fallback to the non-CuDNN implementatation be allowed? If set to false, an exception in CuDNN will be propagated back to the user.
protected double	decay At test time: we can use a global estimate of the mean and variance, calculated using a moving average of the batch means/variances.
protected double	eps Epsilon value for batch normalization; small floating point value added to variance (algorithm 1 in http://arxiv.org/pdf/1502.03167v3.pdf) to reduce/avoid underflow issues. Default: 1e-5
protected double	gamma Used only when 'true' is passed to lockGammaBeta(boolean).
protected List<LayerConstraint>	gammaConstraints Set constraints to be applied to the gamma parameter of this batch normalisation layer.
protected boolean	isMinibatch If doing minibatch training or not.
protected boolean	lockGammaBeta If set to true: lock the gamma and beta parameters to the values for each activation, specified by gamma(double) and beta(double).
protected boolean	useLogStd How should the moving average of variance be stored? Two different parameterizations are supported.

Fields inherited from class org.deeplearning4j.nn.conf.layers.FeedForwardLayer.Builder

nIn, nOut

Fields inherited from class org.deeplearning4j.nn.conf.layers.BaseLayer.Builder

activationFn, biasInit, biasUpdater, gainInit, gradientNormalization, gradientNormalizationThreshold, iupdater, regularization, regularizationBias, weightInitFn, weightNoise

Fields inherited from class org.deeplearning4j.nn.conf.layers.Layer.Builder

allParamConstraints, biasConstraints, iDropout, layerName, weightConstraints

Constructor Summary

Constructors

Constructor and Description
Builder()
Builder(boolean lockGammaBeta)
Builder(double decay, boolean isMinibatch)
Builder(double gamma, double beta)
Builder(double gamma, double beta, boolean lockGammaBeta)

Method Summary

Modifier and Type	Method and Description
BatchNormalization.Builder	beta(double beta) Used only when 'true' is passed to lockGammaBeta(boolean).
BatchNormalization	build()
BatchNormalization.Builder	constrainBeta(LayerConstraint... constraints) Set constraints to be applied to the beta parameter of this batch normalisation layer.
BatchNormalization.Builder	constrainGamma(LayerConstraint... constraints) Set constraints to be applied to the gamma parameter of this batch normalisation layer.
BatchNormalization.Builder	cudnnAllowFallback(boolean allowFallback) When using CuDNN and an error is encountered, should fallback to the non-CuDNN implementatation be allowed? If set to false, an exception in CuDNN will be propagated back to the user.
BatchNormalization.Builder	decay(double decay) At test time: we can use a global estimate of the mean and variance, calculated using a moving average of the batch means/variances.
BatchNormalization.Builder	eps(double eps) Epsilon value for batch normalization; small floating point value added to variance (algorithm 1 in http://arxiv.org/pdf/1502.03167v3.pdf) to reduce/avoid underflow issues. Default: 1e-5
BatchNormalization.Builder	gamma(double gamma) Used only when 'true' is passed to lockGammaBeta(boolean).
BatchNormalization.Builder	lockGammaBeta(boolean lockGammaBeta) If set to true: lock the gamma and beta parameters to the values for each activation, specified by gamma(double) and beta(double).
BatchNormalization.Builder	minibatch(boolean minibatch) If doing minibatch training or not.
BatchNormalization.Builder	useLogStd(boolean useLogStd) How should the moving average of variance be stored? Two different parameterizations are supported.

Methods inherited from class org.deeplearning4j.nn.conf.layers.FeedForwardLayer.Builder

nIn, nIn, nOut, nOut, units

Methods inherited from class org.deeplearning4j.nn.conf.layers.BaseLayer.Builder

activation, activation, biasInit, biasUpdater, dist, gainInit, gradientNormalization, gradientNormalizationThreshold, l1, l1Bias, l2, l2Bias, regularization, regularizationBias, updater, updater, weightDecay, weightDecay, weightDecayBias, weightDecayBias, weightInit, weightInit, weightInit, weightNoise

Methods inherited from class org.deeplearning4j.nn.conf.layers.Layer.Builder

constrainAllParameters, constrainBias, constrainWeights, dropOut, dropOut, name

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

decay

protected double decay

At test time: we can use a global estimate of the mean and variance, calculated using a moving average of the batch means/variances. This moving average is implemented as:
globalMeanEstimate = decay * globalMeanEstimate + (1-decay) * batchMean
globalVarianceEstimate = decay * globalVarianceEstimate + (1-decay) * batchVariance

eps

protected double eps

Epsilon value for batch normalization; small floating point value added to variance (algorithm 1 in http://arxiv.org/pdf/1502.03167v3.pdf) to reduce/avoid underflow issues.
Default: 1e-5

isMinibatch

protected boolean isMinibatch

If doing minibatch training or not. Default: true. Under most circumstances, this should be set to true. If doing full batch training (i.e., all examples in a single DataSet object - very small data sets) then this should be set to false. Affects how global mean/variance estimates are calculated.

lockGammaBeta

protected boolean lockGammaBeta

If set to true: lock the gamma and beta parameters to the values for each activation, specified by gamma(double) and beta(double). Default: false -> learn gamma and beta parameter values during network training.

gamma

protected double gamma

Used only when 'true' is passed to lockGammaBeta(boolean). Value is not used otherwise.
Default: 1.0

beta

protected double beta

Used only when 'true' is passed to lockGammaBeta(boolean). Value is not used otherwise.
Default: 0.0

betaConstraints

protected List<LayerConstraint> betaConstraints

Set constraints to be applied to the beta parameter of this batch normalisation layer. Default: no constraints.
Constraints can be used to enforce certain conditions (non-negativity of parameters, max-norm regularization, etc). These constraints are applied at each iteration, after the parameters have been updated.

gammaConstraints

protected List<LayerConstraint> gammaConstraints

Set constraints to be applied to the gamma parameter of this batch normalisation layer. Default: no constraints.
Constraints can be used to enforce certain conditions (non-negativity of parameters, max-norm regularization, etc). These constraints are applied at each iteration, after the parameters have been updated.

cudnnAllowFallback

protected boolean cudnnAllowFallback

When using CuDNN and an error is encountered, should fallback to the non-CuDNN implementatation be allowed? If set to false, an exception in CuDNN will be propagated back to the user. If false, the built-in (non-CuDNN) implementation for BatchNormalization will be used

useLogStd

protected boolean useLogStd

How should the moving average of variance be stored? Two different parameterizations are supported. useLogStd(false): equivalent to 1.0.0-beta3 and earlier. The variance "parameter" is stored directly as variable
useLogStd(true): (Default) variance is stored as log10(stdev)
The motivation here is for numerical stability (FP16 etc) and also distributed training: storing the variance directly can cause numerical issues. For example, a standard deviation of 1e-3 (something that could be encountered in practice) gives a variance of 1e-6, which can be problematic for 16-bit floating point

Constructor Detail

Builder

public Builder(double decay,
               boolean isMinibatch)

Builder

public Builder(double gamma,
               double beta)

Builder

public Builder(double gamma,
               double beta,
               boolean lockGammaBeta)

Builder

public Builder(boolean lockGammaBeta)

Builder

public Builder()

Method Detail

minibatch

public BatchNormalization.Builder minibatch(boolean minibatch)

If doing minibatch training or not. Default: true. Under most circumstances, this should be set to true. If doing full batch training (i.e., all examples in a single DataSet object - very small data sets) then this should be set to false. Affects how global mean/variance estimates are calculated.

Parameters:

minibatch - Minibatch parameter

gamma

public BatchNormalization.Builder gamma(double gamma)

Used only when 'true' is passed to lockGammaBeta(boolean). Value is not used otherwise.
Default: 1.0

Parameters:

gamma - Gamma parameter for all activations, used only with locked gamma/beta configuration mode

beta

public BatchNormalization.Builder beta(double beta)

Used only when 'true' is passed to lockGammaBeta(boolean). Value is not used otherwise.
Default: 0.0

Parameters:

beta - Beta parameter for all activations, used only with locked gamma/beta configuration mode

eps

public BatchNormalization.Builder eps(double eps)

Epsilon value for batch normalization; small floating point value added to variance (algorithm 1 in http://arxiv.org/pdf/1502.03167v3.pdf) to reduce/avoid underflow issues.
Default: 1e-5

Parameters:

eps - Epsilon values to use

decay

public BatchNormalization.Builder decay(double decay)

At test time: we can use a global estimate of the mean and variance, calculated using a moving average of the batch means/variances. This moving average is implemented as:
globalMeanEstimate = decay * globalMeanEstimate + (1-decay) * batchMean
globalVarianceEstimate = decay * globalVarianceEstimate + (1-decay) * batchVariance

Parameters:

decay - Decay value to use for global stats calculation

lockGammaBeta

public BatchNormalization.Builder lockGammaBeta(boolean lockGammaBeta)

If set to true: lock the gamma and beta parameters to the values for each activation, specified by gamma(double) and beta(double). Default: false -> learn gamma and beta parameter values during network training.

Parameters:

lockGammaBeta - If true: use fixed beta/gamma values. False: learn during

constrainBeta

public BatchNormalization.Builder constrainBeta(LayerConstraint... constraints)

Set constraints to be applied to the beta parameter of this batch normalisation layer. Default: no constraints.
Constraints can be used to enforce certain conditions (non-negativity of parameters, max-norm regularization, etc). These constraints are applied at each iteration, after the parameters have been updated.

Parameters:

constraints - Constraints to apply to the beta parameter of this layer

constrainGamma

public BatchNormalization.Builder constrainGamma(LayerConstraint... constraints)

Set constraints to be applied to the gamma parameter of this batch normalisation layer. Default: no constraints.
Constraints can be used to enforce certain conditions (non-negativity of parameters, max-norm regularization, etc). These constraints are applied at each iteration, after the parameters have been updated.

Parameters:

constraints - Constraints to apply to the gamma parameter of this layer

cudnnAllowFallback

public BatchNormalization.Builder cudnnAllowFallback(boolean allowFallback)

When using CuDNN and an error is encountered, should fallback to the non-CuDNN implementatation be allowed? If set to false, an exception in CuDNN will be propagated back to the user. If false, the built-in (non-CuDNN) implementation for BatchNormalization will be used

Parameters:

allowFallback - Whether fallback to non-CuDNN implementation should be used

useLogStd

public BatchNormalization.Builder useLogStd(boolean useLogStd)

How should the moving average of variance be stored? Two different parameterizations are supported. useLogStd(false): equivalent to 1.0.0-beta3 and earlier. The variance "parameter" is stored directly as variable
useLogStd(true): (Default) variance is stored as log10(stdev)
The motivation here is for numerical stability (FP16 etc) and also distributed training: storing the variance directly can cause numerical issues. For example, a standard deviation of 1e-3 (something that could be encountered in practice) gives a variance of 1e-6, which can be problematic for 16-bit floating point

build

public BatchNormalization build()

Specified by:

build in class Layer.Builder<BatchNormalization.Builder>