org.deeplearning4j.nn.weights

Enum WeightInit

java.lang.Object

java.lang.Enum<WeightInit>

org.deeplearning4j.nn.weights.WeightInit

All Implemented Interfaces:

Serializable, Comparable<WeightInit>

public enum WeightInit
extends Enum<WeightInit>

Weight initialization scheme

DISTRIBUTION: Sample weights from a provided distribution

ZERO: Generate weights as zeros

ONES: All weights are set to 1

SIGMOID_UNIFORM: A version of XAVIER_UNIFORM for sigmoid activation functions. U(-r,r) with r=4*sqrt(6/(fanIn + fanOut))

NORMAL: Normal/Gaussian distribution, with mean 0 and standard deviation 1/sqrt(fanIn). This is the initialization recommented in Klambauer et al. 2017, "Self-Normalizing Neural Network". Equivalent to DL4J's XAVIER_FAN_IN and LECUN_NORMAL (i.e. Keras' "lecun_normal")

LECUN_UNIFORM Uniform U[-a,a] with a=3/sqrt(fanIn).

UNIFORM: Uniform U[-a,a] with a=1/sqrt(fanIn). "Commonly used heuristic" as per Glorot and Bengio 2010

XAVIER: As per Glorot and Bengio 2010: Gaussian distribution with mean 0, variance 2.0/(fanIn + fanOut)

XAVIER_UNIFORM: As per Glorot and Bengio 2010: Uniform distribution U(-s,s) with s = sqrt(6/(fanIn + fanOut))

XAVIER_FAN_IN: Similar to Xavier, but 1/fanIn -> Caffe originally used this.

XAVIER_LEGACY: Xavier weight init in DL4J up to 0.6.0. XAVIER should be preferred.

RELU: He et al. (2015), "Delving Deep into Rectifiers". Normal distribution with variance 2.0/nIn

RELU_UNIFORM: He et al. (2015), "Delving Deep into Rectifiers". Uniform distribution U(-s,s) with s = sqrt(6/fanIn)

IDENTITY: Weights are set to an identity matrix. Note: can only be used with square weight matrices

VAR_SCALING_NORMAL_FAN_IN Gaussian distribution with mean 0, variance 1.0/(fanIn)

VAR_SCALING_NORMAL_FAN_OUT Gaussian distribution with mean 0, variance 1.0/(fanOut)

VAR_SCALING_NORMAL_FAN_AVG Gaussian distribution with mean 0, variance 1.0/((fanIn + fanOut)/2)

VAR_SCALING_UNIFORM_FAN_IN Uniform U[-a,a] with a=3.0/(fanIn)

VAR_SCALING_UNIFORM_FAN_OUT Uniform U[-a,a] with a=3.0/(fanOut)

VAR_SCALING_UNIFORM_FAN_AVG Uniform U[-a,a] with a=3.0/((fanIn + fanOut)/2)

Author:

Adam Gibson

Enum Constant Summary

Enum Constants

Enum Constant and Description
DISTRIBUTION
IDENTITY
LECUN_NORMAL
LECUN_UNIFORM
NORMAL
ONES
RELU
RELU_UNIFORM
SIGMOID_UNIFORM
UNIFORM
VAR_SCALING_NORMAL_FAN_AVG
VAR_SCALING_NORMAL_FAN_IN
VAR_SCALING_NORMAL_FAN_OUT
VAR_SCALING_UNIFORM_FAN_AVG
VAR_SCALING_UNIFORM_FAN_IN
VAR_SCALING_UNIFORM_FAN_OUT
XAVIER
XAVIER_FAN_IN
XAVIER_LEGACY
XAVIER_UNIFORM
ZERO

Method Summary

Modifier and Type	Method and Description
static WeightInit	valueOf(String name) Returns the enum constant of this type with the specified name.
static WeightInit[]	values() Returns an array containing the constants of this enum type, in the order they are declared.

Methods inherited from class java.lang.Enum

clone, compareTo, equals, finalize, getDeclaringClass, hashCode, name, ordinal, toString, valueOf

Methods inherited from class java.lang.Object

getClass, notify, notifyAll, wait, wait, wait

Enum Constant Detail

DISTRIBUTION

public static final WeightInit DISTRIBUTION

ZERO

public static final WeightInit ZERO

ONES

public static final WeightInit ONES

SIGMOID_UNIFORM

public static final WeightInit SIGMOID_UNIFORM

NORMAL

public static final WeightInit NORMAL

LECUN_NORMAL

public static final WeightInit LECUN_NORMAL

UNIFORM

public static final WeightInit UNIFORM

XAVIER

public static final WeightInit XAVIER

XAVIER_UNIFORM

public static final WeightInit XAVIER_UNIFORM

XAVIER_FAN_IN

public static final WeightInit XAVIER_FAN_IN

XAVIER_LEGACY

public static final WeightInit XAVIER_LEGACY

RELU

public static final WeightInit RELU

RELU_UNIFORM

public static final WeightInit RELU_UNIFORM

IDENTITY

public static final WeightInit IDENTITY

LECUN_UNIFORM

public static final WeightInit LECUN_UNIFORM

VAR_SCALING_NORMAL_FAN_IN

public static final WeightInit VAR_SCALING_NORMAL_FAN_IN

VAR_SCALING_NORMAL_FAN_OUT

public static final WeightInit VAR_SCALING_NORMAL_FAN_OUT

VAR_SCALING_NORMAL_FAN_AVG

public static final WeightInit VAR_SCALING_NORMAL_FAN_AVG

VAR_SCALING_UNIFORM_FAN_IN

public static final WeightInit VAR_SCALING_UNIFORM_FAN_IN

VAR_SCALING_UNIFORM_FAN_OUT

public static final WeightInit VAR_SCALING_UNIFORM_FAN_OUT

VAR_SCALING_UNIFORM_FAN_AVG

public static final WeightInit VAR_SCALING_UNIFORM_FAN_AVG

Method Detail

values

public static WeightInit[] values()

Returns an array containing the constants of this enum type, in the order they are declared. This method may be used to iterate over the constants as follows:

for (WeightInit c : WeightInit.values())
    System.out.println(c);

Returns:

an array containing the constants of this enum type, in the order they are declared

valueOf

public static WeightInit valueOf(String name)

Returns the enum constant of this type with the specified name. The string must match exactly an identifier used to declare an enum constant in this type. (Extraneous whitespace characters are not permitted.)

Parameters:

name - the name of the enum constant to be returned.

Returns:

the enum constant with the specified name

Throws:

IllegalArgumentException - if this enum type has no constant with the specified name

NullPointerException - if the argument is null