org.deeplearning4j.nn.layers.recurrent

Class LSTMHelpers

java.lang.Object

org.deeplearning4j.nn.layers.recurrent.LSTMHelpers

public class LSTMHelpers
extends Object

RNN tutorial: http://deeplearning4j.org/usingrnns.html READ THIS FIRST if you want to understand what the heck is happening here. Shared code for the standard "forwards" LSTM RNN and the bidirectional LSTM RNN This was extracted from GravesLSTM and refactored into static helper functions. The general reasoning for this was so we only have math in one place, instead of two. Based on Graves: Supervised Sequence Labelling with Recurrent Neural Networks http://www.cs.toronto.edu/~graves/phd.pdf See also for full/vectorized equations (and a comparison to other LSTM variants): Greff et al. 2015, "LSTM: A Search Space Odyssey", pg11.

When 'hasPeepholeConnections' is true, this is the "vanilla" variant in said paper
When 'hasPeepholeConnections' is false, this is the "no peephole" variant
http://arxiv.org/pdf/1503.04069.pdf

Author:

Alex Black (LSTM implementations), Benjamin Joseph (refactoring for bidirectional LSTM)

Method Summary

Modifier and Type	Method and Description
static FwdPassReturn	activateHelper(BaseLayer layer, NeuralNetConfiguration conf, org.nd4j.linalg.activations.IActivation gateActivationFn, org.nd4j.linalg.api.ndarray.INDArray input, org.nd4j.linalg.api.ndarray.INDArray recurrentWeights, org.nd4j.linalg.api.ndarray.INDArray originalInputWeights, org.nd4j.linalg.api.ndarray.INDArray biases, boolean training, org.nd4j.linalg.api.ndarray.INDArray originalPrevOutputActivations, org.nd4j.linalg.api.ndarray.INDArray originalPrevMemCellState, boolean forBackprop, boolean forwards, String inputWeightKey, org.nd4j.linalg.api.ndarray.INDArray maskArray, boolean hasPeepholeConnections, LSTMHelper helper, CacheMode cacheMode) Returns FwdPassReturn object with activations/INDArrays.
static org.nd4j.linalg.primitives.Pair<Gradient,org.nd4j.linalg.api.ndarray.INDArray>	backpropGradientHelper(NeuralNetConfiguration conf, org.nd4j.linalg.activations.IActivation gateActivationFn, org.nd4j.linalg.api.ndarray.INDArray input, org.nd4j.linalg.api.ndarray.INDArray recurrentWeights, org.nd4j.linalg.api.ndarray.INDArray inputWeights, org.nd4j.linalg.api.ndarray.INDArray epsilon, boolean truncatedBPTT, int tbpttBackwardLength, FwdPassReturn fwdPass, boolean forwards, String inputWeightKey, String recurrentWeightKey, String biasWeightKey, Map<String,org.nd4j.linalg.api.ndarray.INDArray> gradientViews, org.nd4j.linalg.api.ndarray.INDArray maskArray, boolean hasPeepholeConnections, LSTMHelper helper)
static LayerMemoryReport	getMemoryReport(AbstractLSTM lstmLayer, InputType inputType)
static LayerMemoryReport	getMemoryReport(boolean isGraves, FeedForwardLayer lstmLayer, InputType inputType)
static LayerMemoryReport	getMemoryReport(GravesBidirectionalLSTM lstmLayer, InputType inputType)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

activateHelper

public static FwdPassReturn activateHelper(BaseLayer layer,
                                           NeuralNetConfiguration conf,
                                           org.nd4j.linalg.activations.IActivation gateActivationFn,
                                           org.nd4j.linalg.api.ndarray.INDArray input,
                                           org.nd4j.linalg.api.ndarray.INDArray recurrentWeights,
                                           org.nd4j.linalg.api.ndarray.INDArray originalInputWeights,
                                           org.nd4j.linalg.api.ndarray.INDArray biases,
                                           boolean training,
                                           org.nd4j.linalg.api.ndarray.INDArray originalPrevOutputActivations,
                                           org.nd4j.linalg.api.ndarray.INDArray originalPrevMemCellState,
                                           boolean forBackprop,
                                           boolean forwards,
                                           String inputWeightKey,
                                           org.nd4j.linalg.api.ndarray.INDArray maskArray,
                                           boolean hasPeepholeConnections,
                                           LSTMHelper helper,
                                           CacheMode cacheMode)

Returns FwdPassReturn object with activations/INDArrays. Allows activateHelper to be used for forward pass, backward pass and rnnTimeStep whilst being reasonably efficient for all

backpropGradientHelper

public static org.nd4j.linalg.primitives.Pair<Gradient,org.nd4j.linalg.api.ndarray.INDArray> backpropGradientHelper(NeuralNetConfiguration conf,
                                                                                                                    org.nd4j.linalg.activations.IActivation gateActivationFn,
                                                                                                                    org.nd4j.linalg.api.ndarray.INDArray input,
                                                                                                                    org.nd4j.linalg.api.ndarray.INDArray recurrentWeights,
                                                                                                                    org.nd4j.linalg.api.ndarray.INDArray inputWeights,
                                                                                                                    org.nd4j.linalg.api.ndarray.INDArray epsilon,
                                                                                                                    boolean truncatedBPTT,
                                                                                                                    int tbpttBackwardLength,
                                                                                                                    FwdPassReturn fwdPass,
                                                                                                                    boolean forwards,
                                                                                                                    String inputWeightKey,
                                                                                                                    String recurrentWeightKey,
                                                                                                                    String biasWeightKey,
                                                                                                                    Map<String,org.nd4j.linalg.api.ndarray.INDArray> gradientViews,
                                                                                                                    org.nd4j.linalg.api.ndarray.INDArray maskArray,
                                                                                                                    boolean hasPeepholeConnections,
                                                                                                                    LSTMHelper helper)

getMemoryReport

public static LayerMemoryReport getMemoryReport(AbstractLSTM lstmLayer,
                                                InputType inputType)

getMemoryReport

public static LayerMemoryReport getMemoryReport(GravesBidirectionalLSTM lstmLayer,
                                                InputType inputType)

getMemoryReport

public static LayerMemoryReport getMemoryReport(boolean isGraves,
                                                FeedForwardLayer lstmLayer,
                                                InputType inputType)