org.deeplearning4j.nn.updater

Class BaseMultiLayerUpdater<T extends Model>

java.lang.Object

org.deeplearning4j.nn.updater.BaseMultiLayerUpdater<T>

All Implemented Interfaces:

Serializable, Updater

Direct Known Subclasses:

ComputationGraphUpdater, LayerUpdater, MultiLayerUpdater

public abstract class BaseMultiLayerUpdater<T extends Model>
extends Object
implements Updater

BaseMultiLayerUpdater - core functionality for applying updaters to MultiLayerNetwork and ComputationGraph.

This implements updater combining: that is, for any layers (and variables) that:
(a) have contiguous parameters/gradients in the view arrays, and
(b) have identical updater configuration (including updater, LR, LR/momentum schedules, etc - different L1/L2 are OK, however)
are combined into a single GradientUpdater operation, instead of having a set of smaller operations. A smaller number of larger operations improves performance, especially for GPUs.

Author:

Alex Black

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
protected Map<String,Layer>	layersByName
protected T	network
protected List<UpdaterBlock>	updaterBlocks
protected org.nd4j.linalg.api.ndarray.INDArray	updaterStateViewArray

Constructor Summary

Constructors

Constructor and Description
BaseMultiLayerUpdater(T network)
BaseMultiLayerUpdater(T network, org.nd4j.linalg.api.ndarray.INDArray updaterState)

Method Summary

Modifier and Type	Method and Description
boolean	equals(Object o)
protected abstract org.nd4j.linalg.api.ndarray.INDArray	getFlattenedGradientsView()
protected abstract Layer[]	getOrderedLayers()
protected abstract org.nd4j.linalg.api.ndarray.INDArray	getParams()
org.nd4j.linalg.api.ndarray.INDArray	getStateViewArray()
int	hashCode()
protected abstract boolean	isMiniBatch()
protected boolean	isSingleLayerUpdater()
void	preApply(Layer layer, Gradient gradient, int iteration) Pre-apply: Apply gradient normalization/clipping
void	setStateViewArray(org.nd4j.linalg.api.ndarray.INDArray viewArray) Set the view array.
void	setStateViewArray(Layer layer, org.nd4j.linalg.api.ndarray.INDArray viewArray, boolean initialize) Set the internal (historical) state view array for this updater
void	update(Gradient gradient, int iteration, int epoch, int batchSize, LayerWorkspaceMgr workspaceMgr) Update the gradient for the model.
void	update(Layer layer, Gradient gradient, int iteration, int epoch, int batchSize, LayerWorkspaceMgr workspaceMgr) Updater: updates the model

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, toString, wait, wait, wait

Field Detail

network

protected final T extends Model network

layersByName

protected Map<String,Layer> layersByName

updaterBlocks

protected final List<UpdaterBlock> updaterBlocks

updaterStateViewArray

protected org.nd4j.linalg.api.ndarray.INDArray updaterStateViewArray

Constructor Detail

BaseMultiLayerUpdater

public BaseMultiLayerUpdater(T network)

BaseMultiLayerUpdater

public BaseMultiLayerUpdater(T network,
                             org.nd4j.linalg.api.ndarray.INDArray updaterState)

Parameters:

network - Network to create the updater for

updaterState - The updater state to use. Note: This array is used *directly* and isn't copied/cloned

Method Detail

getOrderedLayers

protected abstract Layer[] getOrderedLayers()

Returns:

Array of layers, in the correct order (i.e., same order as the parameter/gradient/updater flattening order - input to output for MultiLayerNetwork, or topological order for ComputationGraph)

getFlattenedGradientsView

protected abstract org.nd4j.linalg.api.ndarray.INDArray getFlattenedGradientsView()

Returns:

The flattened gradient view array for the model

getParams

protected abstract org.nd4j.linalg.api.ndarray.INDArray getParams()

Returns:

The flattened parameter array for the model

isMiniBatch

protected abstract boolean isMiniBatch()

Returns:

True if the configuration for the model is set to minibatch (divide by minibatch size), false otherwise

setStateViewArray

public void setStateViewArray(org.nd4j.linalg.api.ndarray.INDArray viewArray)

Set the view array. Note that this does an assign operation - the provided array is not stored internally.

Parameters:

viewArray - The new updater state

setStateViewArray

public void setStateViewArray(Layer layer,
                              org.nd4j.linalg.api.ndarray.INDArray viewArray,
                              boolean initialize)

Description copied from interface: Updater

Set the internal (historical) state view array for this updater

Specified by:

setStateViewArray in interface Updater

Parameters:

layer - Layer that this updater belongs to

viewArray - View array

initialize - Whether to initialize the array or not

getStateViewArray

public org.nd4j.linalg.api.ndarray.INDArray getStateViewArray()

Specified by:

getStateViewArray in interface Updater

Returns:

the view array for this updater

update

public void update(Layer layer,
                   Gradient gradient,
                   int iteration,
                   int epoch,
                   int batchSize,
                   LayerWorkspaceMgr workspaceMgr)

Description copied from interface: Updater

Updater: updates the model

Specified by:

update in interface Updater

update

public void update(Gradient gradient,
                   int iteration,
                   int epoch,
                   int batchSize,
                   LayerWorkspaceMgr workspaceMgr)

Update the gradient for the model. This operates in 3 steps: 1. Pre-apply: gradient clipping, etc on a per-layer basis 2. Execute the updater (Adam, Nesterov momentum, etc) - in blocks of layers at a time 3. Divide by minibatch size

Parameters:

gradient - Gradient to updater

iteration - The current iteration (i.e., number of parameter updates so far)

batchSize - The current minibatch size (number of examples)

isSingleLayerUpdater

protected boolean isSingleLayerUpdater()

preApply

public void preApply(Layer layer,
                     Gradient gradient,
                     int iteration)

Pre-apply: Apply gradient normalization/clipping

Parameters:

layer - Layer to apply gradient normalization/clipping for

gradient - Gradient to update

iteration - The current iteration (i.e., number of parameter updates so far)

equals

public boolean equals(Object o)

Overrides:

equals in class Object

hashCode

public int hashCode()

Overrides:

hashCode in class Object