org.deeplearning4j.nn.layers

Class BaseLayer<LayerConfT extends BaseLayer>

java.lang.Object

org.deeplearning4j.nn.layers.AbstractLayer<LayerConfT>

org.deeplearning4j.nn.layers.BaseLayer<LayerConfT>

All Implemented Interfaces:

Serializable, Cloneable, Layer, Model, Trainable

Direct Known Subclasses:

BaseOutputLayer, BasePretrainNetwork, BaseRecurrentLayer, BatchNormalization, Cnn3DLossLayer, CnnLossLayer, ConvolutionLayer, DenseLayer, DropoutLayer, ElementWiseMultiplicationLayer, EmbeddingLayer, EmbeddingSequenceLayer, LossLayer, PReLU, RnnLossLayer

public abstract class BaseLayer<LayerConfT extends BaseLayer>
extends AbstractLayer<LayerConfT>

A layer with parameters

Author:

Adam Gibson

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from interface org.deeplearning4j.nn.api.Layer

Layer.TrainingMode, Layer.Type

Field Summary

Fields

Modifier and Type	Field and Description
protected Gradient	gradient
protected INDArray	gradientsFlattened
protected Map<String,INDArray>	gradientViews
protected ConvexOptimizer	optimizer
protected Map<String,INDArray>	params
protected INDArray	paramsFlattened
protected double	score
protected Solver	solver
protected Map<String,INDArray>	weightNoiseParams

Fields inherited from class org.deeplearning4j.nn.layers.AbstractLayer

cacheMode, conf, dataType, dropoutApplied, epochCount, index, input, inputModificationAllowed, iterationCount, maskArray, maskState, preOutput, trainingListeners

Constructor Summary

Constructors

Constructor and Description
BaseLayer(NeuralNetConfiguration conf, org.nd4j.linalg.api.buffer.DataType dataType)

Method Summary

Modifier and Type	Method and Description
INDArray	activate(boolean training, LayerWorkspaceMgr workspaceMgr) Perform forward pass and return the activations array with the last set input
Pair<Gradient,INDArray>	backpropGradient(INDArray epsilon, LayerWorkspaceMgr workspaceMgr) Calculate the gradient relative to the error in the next layer
double	calcRegularizationScore(boolean backpropParamsOnly) Calculate the regularization component of the score, for the parameters in this layer For example, the L1, L2 and/or weight decay components of the loss function
void	clear() Clear input
void	clearNoiseWeightParams()
Layer	clone()
void	computeGradientAndScore(LayerWorkspaceMgr workspaceMgr) Update the score
void	fit() All models have a fit method
void	fit(INDArray input, LayerWorkspaceMgr workspaceMgr) Fit the model to the given data
INDArray	getGradientsViewArray()
ConvexOptimizer	getOptimizer() Returns this models optimizer
INDArray	getParam(String param) Get the parameter
protected INDArray	getParamWithNoise(String param, boolean training, LayerWorkspaceMgr workspaceMgr) Get the parameter, after applying any weight noise (such as DropConnect) if necessary.
Gradient	gradient() Get the gradient.
boolean	hasBias() Does this layer have no bias term? Many layers (dense, convolutional, output, embedding) have biases by default, but no-bias versions are possible via configuration
boolean	hasLayerNorm() Does this layer support and is it enabled layer normalization? Only Dense and SimpleRNN Layers support layer normalization.
LayerConfT	layerConf()
long	numParams() The number of parameters for the model
INDArray	params() Returns the parameters of the neural network as a flattened row vector
Map<String,INDArray>	paramTable() The param table
Map<String,INDArray>	paramTable(boolean backpropParamsOnly) Table of parameters by key, for backprop For many models (dense layers, etc) - all parameters are backprop parameters
protected INDArray	preOutput(boolean training, LayerWorkspaceMgr workspaceMgr)
protected Pair<INDArray,INDArray>	preOutputWithPreNorm(boolean training, boolean forBackprop, LayerWorkspaceMgr workspaceMgr)
double	score() Objective function: the specified objective
void	setBackpropGradientsViewArray(INDArray gradients) Set the gradients array as a view of the full (backprop) network parameters NOTE: this is intended to be used internally in MultiLayerNetwork and ComputationGraph, not by users.
void	setParam(String key, INDArray val) Set the parameter with a new ndarray
void	setParams(INDArray params) Set the parameters for this model.
protected void	setParams(INDArray params, char order)
void	setParamsViewArray(INDArray params) Set the initial parameters array as a view of the full (backprop) network parameters NOTE: this is intended to be used internally in MultiLayerNetwork and ComputationGraph, not by users.
void	setParamTable(Map<String,INDArray> paramTable) Setter for the param table
protected void	setScoreWithZ(INDArray z)
String	toString()
void	update(Gradient gradient) Update layer weights and biases with gradient change
void	update(INDArray gradient, String paramType) Perform one update applying the gradient

Methods inherited from class org.deeplearning4j.nn.layers.AbstractLayer

activate, addListeners, allowInputModification, applyConstraints, applyDropOutIfNecessary, applyMask, assertInputSet, backpropDropOutIfPresent, batchSize, conf, feedForwardMaskArray, getConfig, getEpochCount, getHelper, getIndex, getInput, getInputMiniBatchSize, getListeners, getMaskArray, gradientAndScore, init, input, layerId, numParams, setCacheMode, setConf, setEpochCount, setIndex, setInput, setInputMiniBatchSize, setListeners, setListeners, setMaskArray, type, updaterDivideByMinibatch

Methods inherited from class java.lang.Object

equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.deeplearning4j.nn.api.Layer

getIterationCount, isPretrainLayer, setIterationCount

Field Detail

paramsFlattened

protected INDArray paramsFlattened

gradientsFlattened

protected INDArray gradientsFlattened

params

protected Map<String,INDArray> params

gradientViews

protected transient Map<String,INDArray> gradientViews

score

protected double score

optimizer

protected ConvexOptimizer optimizer

gradient

protected Gradient gradient

solver

protected Solver solver

weightNoiseParams

protected Map<String,INDArray> weightNoiseParams

Constructor Detail

BaseLayer

public BaseLayer(NeuralNetConfiguration conf,
                 org.nd4j.linalg.api.buffer.DataType dataType)

Method Detail

layerConf

public LayerConfT layerConf()

Overrides:

layerConf in class AbstractLayer<LayerConfT extends BaseLayer>

backpropGradient

public Pair<Gradient,INDArray> backpropGradient(INDArray epsilon,
                                                LayerWorkspaceMgr workspaceMgr)

Description copied from interface: Layer

Calculate the gradient relative to the error in the next layer

Parameters:

epsilon - w^(L+1)*delta^(L+1). Or, equiv: dC/da, i.e., (dC/dz)*(dz/da) = dC/da, where C is cost function a=sigma(z) is activation.

workspaceMgr - Workspace manager

Returns:

Pair where Gradient is gradient for this layer, INDArray is epsilon (activation gradient) needed by next layer, but before element-wise multiply by sigmaPrime(z). So for standard feed-forward layer, if this layer is L, then return.getSecond() == dL/dIn = (w^(L)*(delta^(L))^T)^T. Note that the returned array should be placed in the ArrayType.ACTIVATION_GRAD workspace via the workspace manager

fit

public void fit()

Description copied from interface: Model

All models have a fit method

Specified by:

fit in interface Model

Overrides:

fit in class AbstractLayer<LayerConfT extends BaseLayer>

computeGradientAndScore

public void computeGradientAndScore(LayerWorkspaceMgr workspaceMgr)

Description copied from interface: Model

Update the score

Specified by:

computeGradientAndScore in interface Model

Overrides:

computeGradientAndScore in class AbstractLayer<LayerConfT extends BaseLayer>

setScoreWithZ

protected void setScoreWithZ(INDArray z)

score

public double score()

Objective function: the specified objective

Specified by:

score in interface Model

Overrides:

score in class AbstractLayer<LayerConfT extends BaseLayer>

Returns:

the score for the objective

gradient

public Gradient gradient()

Description copied from interface: Model

Get the gradient. Note that this method will not calculate the gradient, it will rather return the gradient that has been computed before. For calculating the gradient, see Model.computeGradientAndScore(LayerWorkspaceMgr) } .

Specified by:

gradient in interface Model

Overrides:

gradient in class AbstractLayer<LayerConfT extends BaseLayer>

Returns:

the gradient for this model, as calculated before

update

public void update(Gradient gradient)

Description copied from interface: Model

Update layer weights and biases with gradient change

Specified by:

update in interface Model

Overrides:

update in class AbstractLayer<LayerConfT extends BaseLayer>

update

public void update(INDArray gradient,
                   String paramType)

Description copied from interface: Model

Perform one update applying the gradient

Specified by:

update in interface Model

Overrides:

update in class AbstractLayer<LayerConfT extends BaseLayer>

Parameters:

gradient - the gradient to apply

getOptimizer

public ConvexOptimizer getOptimizer()

Description copied from interface: Model

Returns this models optimizer

Specified by:

getOptimizer in interface Model

Overrides:

getOptimizer in class AbstractLayer<LayerConfT extends BaseLayer>

Returns:

this models optimizer

params

public INDArray params()

Returns the parameters of the neural network as a flattened row vector

Specified by:

params in interface Model

Specified by:

params in interface Trainable

Overrides:

params in class AbstractLayer<LayerConfT extends BaseLayer>

Returns:

the parameters of the neural network

getParam

public INDArray getParam(String param)

Description copied from interface: Model

Get the parameter

Specified by:

getParam in interface Model

Overrides:

getParam in class AbstractLayer<LayerConfT extends BaseLayer>

Parameters:

param - the key of the parameter

Returns:

the parameter vector/matrix with that particular key

setParam

public void setParam(String key,
                     INDArray val)

Description copied from interface: Model

Set the parameter with a new ndarray

Specified by:

setParam in interface Model

Overrides:

setParam in class AbstractLayer<LayerConfT extends BaseLayer>

Parameters:

key - the key to se t

val - the new ndarray

setParams

public void setParams(INDArray params)

Description copied from interface: Model

Set the parameters for this model. This expects a linear ndarray which then be unpacked internally relative to the expected ordering of the model

Specified by:

setParams in interface Model

Overrides:

setParams in class AbstractLayer<LayerConfT extends BaseLayer>

Parameters:

params - the parameters for the model

setParams

protected void setParams(INDArray params,
                         char order)

Overrides:

setParams in class AbstractLayer<LayerConfT extends BaseLayer>

setParamsViewArray

public void setParamsViewArray(INDArray params)

Description copied from interface: Model

Set the initial parameters array as a view of the full (backprop) network parameters NOTE: this is intended to be used internally in MultiLayerNetwork and ComputationGraph, not by users.

Specified by:

setParamsViewArray in interface Model

Overrides:

setParamsViewArray in class AbstractLayer<LayerConfT extends BaseLayer>

Parameters:

params - a 1 x nParams row vector that is a view of the larger (MLN/CG) parameters array

getGradientsViewArray

public INDArray getGradientsViewArray()

Specified by:

getGradientsViewArray in interface Model

Specified by:

getGradientsViewArray in interface Trainable

Overrides:

getGradientsViewArray in class AbstractLayer<LayerConfT extends BaseLayer>

Returns:

1D gradients view array

setBackpropGradientsViewArray

public void setBackpropGradientsViewArray(INDArray gradients)

Description copied from interface: Model

Set the gradients array as a view of the full (backprop) network parameters NOTE: this is intended to be used internally in MultiLayerNetwork and ComputationGraph, not by users.

Specified by:

setBackpropGradientsViewArray in interface Model

Overrides:

setBackpropGradientsViewArray in class AbstractLayer<LayerConfT extends BaseLayer>

Parameters:

gradients - a 1 x nParams row vector that is a view of the larger (MLN/CG) gradients array

setParamTable

public void setParamTable(Map<String,INDArray> paramTable)

Description copied from interface: Model

Setter for the param table

Specified by:

setParamTable in interface Model

Overrides:

setParamTable in class AbstractLayer<LayerConfT extends BaseLayer>

paramTable

public Map<String,INDArray> paramTable()

Description copied from interface: Model

The param table

Specified by:

paramTable in interface Model

Overrides:

paramTable in class AbstractLayer<LayerConfT extends BaseLayer>

Returns:

paramTable

public Map<String,INDArray> paramTable(boolean backpropParamsOnly)

Description copied from interface: Model

Table of parameters by key, for backprop For many models (dense layers, etc) - all parameters are backprop parameters

Specified by:

paramTable in interface Model

Specified by:

paramTable in interface Trainable

Overrides:

paramTable in class AbstractLayer<LayerConfT extends BaseLayer>

Parameters:

backpropParamsOnly - If true, return backprop params only. If false: return all params (equivalent to paramsTable())

Returns:

Parameter table

getParamWithNoise

protected INDArray getParamWithNoise(String param,
                                     boolean training,
                                     LayerWorkspaceMgr workspaceMgr)

Get the parameter, after applying any weight noise (such as DropConnect) if necessary. Note that during training, this will store the post-noise parameters, as these should be used for both forward pass and backprop, for a single iteration. Consequently, the parameters (post noise) should be cleared after each training iteration

Parameters:

param - Parameter key

training - If true: during training

Returns:

The parameter, after applying any noise

preOutput

protected INDArray preOutput(boolean training,
                             LayerWorkspaceMgr workspaceMgr)

preOutputWithPreNorm

protected Pair<INDArray,INDArray> preOutputWithPreNorm(boolean training,
                                                       boolean forBackprop,
                                                       LayerWorkspaceMgr workspaceMgr)

activate

public INDArray activate(boolean training,
                         LayerWorkspaceMgr workspaceMgr)

Description copied from interface: Layer

Perform forward pass and return the activations array with the last set input

Parameters:

training - training or test mode

workspaceMgr - Workspace manager

Returns:

the activation (layer output) of the last specified input. Note that the returned array should be placed in the ArrayType.ACTIVATIONS workspace via the workspace manager

calcRegularizationScore

public double calcRegularizationScore(boolean backpropParamsOnly)

Description copied from interface: Layer

Calculate the regularization component of the score, for the parameters in this layer
For example, the L1, L2 and/or weight decay components of the loss function

Specified by:

calcRegularizationScore in interface Layer

Overrides:

calcRegularizationScore in class AbstractLayer<LayerConfT extends BaseLayer>

Parameters:

backpropParamsOnly - If true: calculate regularization score based on backprop params only. If false: calculate based on all params (including pretrain params, if any)

Returns:

the regularization score of

clone

public Layer clone()

Overrides:

clone in class Object

numParams

public long numParams()

The number of parameters for the model

Specified by:

numParams in interface Model

Specified by:

numParams in interface Trainable

Overrides:

numParams in class AbstractLayer<LayerConfT extends BaseLayer>

Returns:

the number of parameters for the model

fit

public void fit(INDArray input,
                LayerWorkspaceMgr workspaceMgr)

Description copied from interface: Model

Fit the model to the given data

Specified by:

fit in interface Model

Overrides:

fit in class AbstractLayer<LayerConfT extends BaseLayer>

Parameters:

input - the data to fit the model to

toString

public String toString()

Overrides:

toString in class Object

clear

public void clear()

Description copied from interface: Model

Clear input

Specified by:

clear in interface Model

Overrides:

clear in class AbstractLayer<LayerConfT extends BaseLayer>

clearNoiseWeightParams

public void clearNoiseWeightParams()

hasBias

public boolean hasBias()

Does this layer have no bias term? Many layers (dense, convolutional, output, embedding) have biases by default, but no-bias versions are possible via configuration

Returns:

True if a bias term is present, false otherwise

hasLayerNorm

public boolean hasLayerNorm()

Does this layer support and is it enabled layer normalization? Only Dense and SimpleRNN Layers support layer normalization.

Returns:

True if layer normalization is enabled on this layer, false otherwise