org.deeplearning4j.nn.conf.dropout

Class Dropout

java.lang.Object

org.deeplearning4j.nn.conf.dropout.Dropout

All Implemented Interfaces:

Serializable, Cloneable, IDropout

public class Dropout
extends Object
implements IDropout

Implements standard (inverted) dropout.

Regarding dropout probability. This is the probability of retaining each input activation value for a layer. Thus, each input activation x is independently set to:
x <- 0, with probability 1-p
x <- x/p with probability p
Note that this "inverted" dropout scheme maintains the expected value of activations - i.e., E(x) is the same before and after dropout.
Dropout schedules (i.e., varying probability p as a function of iteration/epoch) are also supported.

Other libraries (notably, Keras) use p == probability(dropping an activation)
In DL4J, new Dropout(x) will keep an input activation with probability x, and set to 0 with probability 1-x.
Thus, a dropout value of 1.0 is functionally equivalent to no dropout: i.e., 100% probability of retaining each input activation.

Note 1: As per all IDropout instances, dropout is applied at training time only - and is automatically not applied at test time (for evaluation, etc)
Note 2: Care should be taken when setting lower (probability of retaining) values for (too much information may be lost with aggressive (very low) dropout values).
Note 3: Frequently, dropout is not applied to (or, has higher retain probability for) input (first layer) layers. Dropout is also often not applied to output layers.
Note 4: Implementation detail (most users can ignore): DL4J uses inverted dropout, as described here: http://cs231n.github.io/neural-networks-2/

See: Srivastava et al. 2014: Dropout: A Simple Way to Prevent Neural Networks from Overfitting http://www.cs.toronto.edu/~rsalakhu/papers/srivastava14a.pdf

Author:

Alex Black

See Also:

Serialized Form

Constructor Summary

Constructors

Modifier	Constructor and Description
	Dropout(double activationRetainProbability)
protected	Dropout(double activationRetainProbability, ISchedule activationRetainProbabilitySchedule)
	Dropout(ISchedule activationRetainProbabilitySchedule)

Method Summary

Modifier and Type	Method and Description
INDArray	applyDropout(INDArray inputActivations, INDArray output, int iteration, int epoch, LayerWorkspaceMgr workspaceMgr)
INDArray	backprop(INDArray gradAtOutput, INDArray gradAtInput, int iteration, int epoch) Perform backprop.
void	clear() Clear the internal state (for example, dropout mask) if any is present
Dropout	clone()
protected void	initializeHelper() Initialize the CuDNN dropout helper, if possible

Methods inherited from class java.lang.Object

equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

Dropout

public Dropout(double activationRetainProbability)

Parameters:

activationRetainProbability - Probability of retaining an activation - see Dropout javadoc

Dropout

public Dropout(ISchedule activationRetainProbabilitySchedule)

Parameters:

activationRetainProbabilitySchedule - Schedule for probability of retaining an activation - see Dropout javadoc

Dropout

protected Dropout(double activationRetainProbability,
                  ISchedule activationRetainProbabilitySchedule)

Method Detail

initializeHelper

protected void initializeHelper()

Initialize the CuDNN dropout helper, if possible

applyDropout

public INDArray applyDropout(INDArray inputActivations,
                             INDArray output,
                             int iteration,
                             int epoch,
                             LayerWorkspaceMgr workspaceMgr)

Specified by:

applyDropout in interface IDropout

Parameters:

inputActivations - Input activations array

output - The result array (same as inputArray for in-place ops) for the post-dropout activations

iteration - Current iteration number

epoch - Current epoch number

workspaceMgr - Workspace manager, if any storage is required (use ArrayType.INPUT)

Returns:

The output (resultArray) after applying dropout

backprop

public INDArray backprop(INDArray gradAtOutput,
                         INDArray gradAtInput,
                         int iteration,
                         int epoch)

Description copied from interface: IDropout

Perform backprop. This should also clear the internal state (dropout mask) if any is present

Specified by:

backprop in interface IDropout

Parameters:

gradAtOutput - Gradients at the output of the dropout op - i.e., dL/dOut

gradAtInput - Gradients at the input of the dropout op - i.e., dL/dIn. Use the same array as gradAtOutput to apply the backprop gradient in-place

iteration - Current iteration

epoch - Current epoch

Returns:

Same array as gradAtInput - i.e., gradient after backpropagating through dropout op - i.e., dL/dIn

clear

public void clear()

Description copied from interface: IDropout

Clear the internal state (for example, dropout mask) if any is present

Specified by:

clear in interface IDropout

clone

public Dropout clone()

Specified by:

clone in interface IDropout

Overrides:

clone in class Object