org.deeplearning4j.nn.conf.dropout

Class AlphaDropout

java.lang.Object

org.deeplearning4j.nn.conf.dropout.AlphaDropout

All Implemented Interfaces:

Serializable, Cloneable, IDropout

public class AlphaDropout
extends Object
implements IDropout

AlphaDropout is a dropout technique proposed by Klaumbauer et al. 2017 - Self-Normalizing Neural Networks https://arxiv.org/abs/1706.02515

This dropout technique was designed specifically for self-normalizing neural networks - i.e., networks using ActivationSELU / Activation.SELU activation function, combined with the N(0,stdev=1/sqrt(fanIn)) "SNN" weight initialization, WeightInit.NORMAL

In conjuction with the aforementioned activation function and weight initialization, AlphaDropout attempts to keep both the mean and variance of the post-dropout activations to the the same (in expectation) as before alpha dropout was applied.
Specifically, AlphaDropout implements a * (x * d + alphaPrime * (1-d)) + b, where d ~ Bernoulli(p), i.e., d \in {0,1}. Where x is the input activations, a, b, alphaPrime are constants determined from the SELU alpha/lambda parameters. Users should use the default alpha/lambda values in virtually all cases.

Dropout schedules (i.e., varying probability p as a function of iteration/epoch) are also supported.

Author:

Alex Black

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
static double	DEFAULT_ALPHA
static double	DEFAULT_LAMBDA

Constructor Summary

Constructors

Modifier	Constructor and Description
	AlphaDropout(double activationRetainProbability)
protected	AlphaDropout(double activationRetainProbability, org.nd4j.linalg.schedule.ISchedule activationRetainProbabilitySchedule, double alpha, double lambda)
	AlphaDropout(org.nd4j.linalg.schedule.ISchedule activationRetainProbabilitySchedule)

Method Summary

Modifier and Type	Method and Description
double	a(double p)
org.nd4j.linalg.api.ndarray.INDArray	applyDropout(org.nd4j.linalg.api.ndarray.INDArray inputActivations, int iteration, int epoch, boolean inPlace)
double	b(double p)
AlphaDropout	clone()

Methods inherited from class java.lang.Object

equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

DEFAULT_ALPHA

public static final double DEFAULT_ALPHA

See Also:

Constant Field Values

DEFAULT_LAMBDA

public static final double DEFAULT_LAMBDA

See Also:

Constant Field Values

Constructor Detail

AlphaDropout

public AlphaDropout(double activationRetainProbability)

Parameters:

activationRetainProbability - Probability of retaining an activation. See AlphaDropout javadoc

AlphaDropout

public AlphaDropout(@NonNull
                    org.nd4j.linalg.schedule.ISchedule activationRetainProbabilitySchedule)

Parameters:

activationRetainProbabilitySchedule - Schedule for the probability of retaining an activation. See AlphaDropout javadoc

AlphaDropout

protected AlphaDropout(double activationRetainProbability,
                       org.nd4j.linalg.schedule.ISchedule activationRetainProbabilitySchedule,
                       double alpha,
                       double lambda)

Method Detail

applyDropout

public org.nd4j.linalg.api.ndarray.INDArray applyDropout(org.nd4j.linalg.api.ndarray.INDArray inputActivations,
                                                         int iteration,
                                                         int epoch,
                                                         boolean inPlace)

Specified by:

applyDropout in interface IDropout

Parameters:

inputActivations - Input activations array

iteration - Current iteration number

epoch - Current epoch number

inPlace - If true: modify the input activations in-place. False: Copy the input activations and apply dropout on the copy instead

Returns:

clone

public AlphaDropout clone()

Specified by:

clone in interface IDropout

Overrides:

clone in class Object

a

public double a(double p)

b

public double b(double p)