Package smile.base.mlp

Class Layer

java.lang.Object
smile.base.mlp.Layer
All Implemented Interfaces:
Serializable
Direct Known Subclasses:
HiddenLayer, InputLayer, OutputLayer
public abstract class Layer extends Object implements Serializable
A layer in the neural network.
See Also:

  • Field Summary

    Fields
    Modifier and Type
    Field
    Description
    protected double[]
    bias
    The bias.
    protected ThreadLocal<double[]>
    biasGradient
    The bias gradient.
    protected ThreadLocal<double[]>
    biasGradientMoment1
    The first moment of bias gradient.
    protected ThreadLocal<double[]>
    biasGradientMoment2
    The second moment of bias gradient.
    protected ThreadLocal<double[]>
    biasUpdate
    The bias update.
    protected final double
    dropout
    The dropout rate.
    protected ThreadLocal<byte[]>
    mask
    The dropout mask.
    protected final int
    n
    The number of neurons in this layer
    protected ThreadLocal<double[]>
    output
    The output vector.
    protected ThreadLocal<double[]>
    outputGradient
    The output gradient.
    protected final int
    p
    The number of input variables.
    protected smile.math.matrix.Matrix
    weight
    The affine transformation matrix.
    protected ThreadLocal<smile.math.matrix.Matrix>
    weightGradient
    The weight gradient.
    protected ThreadLocal<smile.math.matrix.Matrix>
    weightGradientMoment1
    The first moment of weight gradient.
    protected ThreadLocal<smile.math.matrix.Matrix>
    weightGradientMoment2
    The second moment of weight gradient.
    protected ThreadLocal<smile.math.matrix.Matrix>
    weightUpdate
    The weight update.

  • Constructor Summary

    Constructors
    Constructor
    Description
    Layer(int n, int p)
    Constructor.
    Layer(int n, int p, double dropout)
    Constructor.
    Layer(smile.math.matrix.Matrix weight, double[] bias)
    Constructor.
    Layer(smile.math.matrix.Matrix weight, double[] bias, double dropout)
    Constructor.

  • Method Summary

    Modifier and Type
    Method
    Description
    void
    backpopagateDropout()
    Propagates the errors back through the (implicit) dropout layer.
    abstract void
    backpropagate(double[] lowerLayerGradient)
    Propagates the errors back to a lower layer.
    static HiddenLayerBuilder
    builder(String activation, int neurons, double dropout, double param)
    Returns a hidden layer.
    void
    computeGradient(double[] x)
    Computes the parameter gradient for a sample of (mini-)batch.
    void
    computeGradientUpdate(double[] x, double learningRate, double momentum, double decay)
    Computes the parameter gradient and update the weights.
    int
    getInputSize()
    Returns the dimension of input vector (not including bias value).
    int
    getOutputSize()
    Returns the dimension of output vector.
    double[]
    gradient()
    Returns the output gradient vector.
    static LayerBuilder
    input(int neurons)
    Returns an input layer.
    static LayerBuilder
    input(int neurons, double dropout)
    Returns an input layer.
    static HiddenLayerBuilder
    leaky(int neurons)
    Returns a hidden layer with leaky rectified linear activation function.
    static HiddenLayerBuilder
    leaky(int neurons, double dropout)
    Returns a hidden layer with leaky rectified linear activation function.
    static HiddenLayerBuilder
    leaky(int neurons, double dropout, double a)
    Returns a hidden layer with leaky rectified linear activation function.
    static HiddenLayerBuilder
    linear(int neurons)
    Returns a hidden layer with linear activation function.
    static HiddenLayerBuilder
    linear(int neurons, double dropout)
    Returns a hidden layer with linear activation function.
    static OutputLayerBuilder
    mle(int neurons, OutputFunction output)
    Returns an output layer with (log-)likelihood cost function.
    static OutputLayerBuilder
    mse(int neurons, OutputFunction output)
    Returns an output layer with mean squared error cost function.
    static LayerBuilder[]
    of(int k, int p, String spec)
    Returns the layer builders given a string representation such as "Input(10, 0.2)|ReLU(50, 0.5)|Sigmoid(30, 0.5)|...".
    double[]
    output()
    Returns the output vector.
    void
    propagate(double[] x)
    Propagates the signals from a lower layer to this layer.
    void
    propagateDropout()
    Propagates the output signals through the implicit dropout layer.
    static HiddenLayerBuilder
    rectifier(int neurons)
    Returns a hidden layer with rectified linear activation function.
    static HiddenLayerBuilder
    rectifier(int neurons, double dropout)
    Returns a hidden layer with rectified linear activation function.
    static HiddenLayerBuilder
    sigmoid(int neurons)
    Returns a hidden layer with sigmoid activation function.
    static HiddenLayerBuilder
    sigmoid(int neurons, double dropout)
    Returns a hidden layer with sigmoid activation function.
    static HiddenLayerBuilder
    tanh(int neurons)
    Returns a hidden layer with hyperbolic tangent activation function.
    static HiddenLayerBuilder
    tanh(int neurons, double dropout)
    Returns a hidden layer with hyperbolic tangent activation function.
    abstract void
    transform(double[] x)
    The activation or output function.
    void
    update(int m, double learningRate, double momentum, double decay, double rho, double epsilon)
    Adjust network weights by back-propagation algorithm.

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Field Details

    • n

      protected final int n
      The number of neurons in this layer

    • p

      protected final int p
      The number of input variables.

    • dropout

      protected final double dropout
      The dropout rate. Dropout randomly sets input units to 0 with this rate at each step during training time, which helps prevent overfitting.

    • weight

      protected smile.math.matrix.Matrix weight
      The affine transformation matrix.

    • bias

      protected double[] bias
      The bias.

    • output

      protected transient ThreadLocal<double[]> output
      The output vector.

    • outputGradient

      protected transient ThreadLocal<double[]> outputGradient
      The output gradient.

    • weightGradient

      protected transient ThreadLocal<smile.math.matrix.Matrix> weightGradient
      The weight gradient.

    • biasGradient

      protected transient ThreadLocal<double[]> biasGradient
      The bias gradient.

    • weightGradientMoment1

      protected transient ThreadLocal<smile.math.matrix.Matrix> weightGradientMoment1
      The first moment of weight gradient.

    • weightGradientMoment2

      protected transient ThreadLocal<smile.math.matrix.Matrix> weightGradientMoment2
      The second moment of weight gradient.

    • biasGradientMoment1

      protected transient ThreadLocal<double[]> biasGradientMoment1
      The first moment of bias gradient.

    • biasGradientMoment2

      protected transient ThreadLocal<double[]> biasGradientMoment2
      The second moment of bias gradient.

    • weightUpdate

      protected transient ThreadLocal<smile.math.matrix.Matrix> weightUpdate
      The weight update.

    • biasUpdate

      protected transient ThreadLocal<double[]> biasUpdate
      The bias update.

    • mask

      protected transient ThreadLocal<byte[]> mask
      The dropout mask.

  • Constructor Details

    • Layer

      public Layer(int n, int p)
      Constructor. Randomly initialized weights and zero bias.
      Parameters:
      n - the number of neurons.
      p - the number of input variables (not including bias value).

    • Layer

      public Layer(int n, int p, double dropout)
      Constructor. Randomly initialized weights and zero bias.
      Parameters:
      n - the number of neurons.
      p - the number of input variables (not including bias value).
      dropout - the dropout rate.

    • Layer

      public Layer(smile.math.matrix.Matrix weight, double[] bias)
      Constructor.
      Parameters:
      weight - the weight matrix.
      bias - the bias vector.

    • Layer

      public Layer(smile.math.matrix.Matrix weight, double[] bias, double dropout)
      Constructor.
      Parameters:
      weight - the weight matrix.
      bias - the bias vector.
      dropout - the dropout rate.

  • Method Details

    • getOutputSize

      public int getOutputSize()
      Returns the dimension of output vector.
      Returns:
      the dimension of output vector.

    • getInputSize

      public int getInputSize()
      Returns the dimension of input vector (not including bias value).
      Returns:
      the dimension of input vector.

    • output

      public double[] output()
      Returns the output vector.
      Returns:
      the output vector.

    • gradient

      public double[] gradient()
      Returns the output gradient vector.
      Returns:
      the output gradient vector.

    • propagate

      public void propagate(double[] x)
      Propagates the signals from a lower layer to this layer.
      Parameters:
      x - the lower layer signals.

    • propagateDropout

      public void propagateDropout()
      Propagates the output signals through the implicit dropout layer. Dropout randomly sets output units to 0. It should only be applied during training.

    • transform

      public abstract void transform(double[] x)
      The activation or output function.
      Parameters:
      x - the input and output values.

    • backpropagate

      public abstract void backpropagate(double[] lowerLayerGradient)
      Propagates the errors back to a lower layer.
      Parameters:
      lowerLayerGradient - the gradient vector of lower layer.

    • backpopagateDropout

      public void backpopagateDropout()
      Propagates the errors back through the (implicit) dropout layer.

    • computeGradientUpdate

      public void computeGradientUpdate(double[] x, double learningRate, double momentum, double decay)
      Computes the parameter gradient and update the weights.
      Parameters:
      x - the input vector.
      learningRate - the learning rate.
      momentum - the momentum factor.
      decay - weight decay factor.

    • computeGradient

      public void computeGradient(double[] x)
      Computes the parameter gradient for a sample of (mini-)batch.
      Parameters:
      x - the input vector.

    • update

      public void update(int m, double learningRate, double momentum, double decay, double rho, double epsilon)
      Adjust network weights by back-propagation algorithm.
      Parameters:
      m - the size of mini-batch.
      learningRate - the learning rate.
      momentum - the momentum factor.
      decay - weight decay factor.
      rho - RMSProp discounting factor for the history/coming gradient.
      epsilon - a small constant for numerical stability.

    • builder

      public static HiddenLayerBuilder builder(String activation, int neurons, double dropout, double param)
      Returns a hidden layer.
      Parameters:
      activation - the activation function.
      neurons - the number of neurons.
      dropout - the dropout rate.
      param - the optional activation function parameter.
      Returns:
      the layer builder.

    • input

      public static LayerBuilder input(int neurons)
      Returns an input layer.
      Parameters:
      neurons - the number of neurons.
      Returns:
      the layer builder.

    • input

      public static LayerBuilder input(int neurons, double dropout)
      Returns an input layer.
      Parameters:
      neurons - the number of neurons.
      dropout - the dropout rate.
      Returns:
      the layer builder.

    • linear

      public static HiddenLayerBuilder linear(int neurons)
      Returns a hidden layer with linear activation function.
      Parameters:
      neurons - the number of neurons.
      Returns:
      the layer builder.

    • linear

      public static HiddenLayerBuilder linear(int neurons, double dropout)
      Returns a hidden layer with linear activation function.
      Parameters:
      neurons - the number of neurons.
      dropout - the dropout rate.
      Returns:
      the layer builder.

    • rectifier

      public static HiddenLayerBuilder rectifier(int neurons)
      Returns a hidden layer with rectified linear activation function.
      Parameters:
      neurons - the number of neurons.
      Returns:
      the layer builder.

    • rectifier

      public static HiddenLayerBuilder rectifier(int neurons, double dropout)
      Returns a hidden layer with rectified linear activation function.
      Parameters:
      neurons - the number of neurons.
      dropout - the dropout rate.
      Returns:
      the layer builder.

    • leaky

      public static HiddenLayerBuilder leaky(int neurons)
      Returns a hidden layer with leaky rectified linear activation function.
      Parameters:
      neurons - the number of neurons.
      Returns:
      the layer builder.

    • leaky

      public static HiddenLayerBuilder leaky(int neurons, double dropout)
      Returns a hidden layer with leaky rectified linear activation function.
      Parameters:
      neurons - the number of neurons.
      dropout - the dropout rate.
      Returns:
      the layer builder.

    • leaky

      public static HiddenLayerBuilder leaky(int neurons, double dropout, double a)
      Returns a hidden layer with leaky rectified linear activation function.
      Parameters:
      neurons - the number of neurons.
      dropout - the dropout rate.
      a - the parameter of leaky ReLU.
      Returns:
      the layer builder.

    • sigmoid

      public static HiddenLayerBuilder sigmoid(int neurons)
      Returns a hidden layer with sigmoid activation function.
      Parameters:
      neurons - the number of neurons.
      Returns:
      the layer builder.

    • sigmoid

      public static HiddenLayerBuilder sigmoid(int neurons, double dropout)
      Returns a hidden layer with sigmoid activation function.
      Parameters:
      neurons - the number of neurons.
      dropout - the dropout rate.
      Returns:
      the layer builder.

    • tanh

      public static HiddenLayerBuilder tanh(int neurons)
      Returns a hidden layer with hyperbolic tangent activation function.
      Parameters:
      neurons - the number of neurons.
      Returns:
      the layer builder.

    • tanh

      public static HiddenLayerBuilder tanh(int neurons, double dropout)
      Returns a hidden layer with hyperbolic tangent activation function.
      Parameters:
      neurons - the number of neurons.
      dropout - the dropout rate.
      Returns:
      the layer builder.

    • mse

      public static OutputLayerBuilder mse(int neurons, OutputFunction output)
      Returns an output layer with mean squared error cost function.
      Parameters:
      neurons - the number of neurons.
      output - the output function.
      Returns:
      the layer builder.

    • mle

      public static OutputLayerBuilder mle(int neurons, OutputFunction output)
      Returns an output layer with (log-)likelihood cost function.
      Parameters:
      neurons - the number of neurons.
      output - the output function.
      Returns:
      the layer builder.

    • of

      public static LayerBuilder[] of(int k, int p, String spec)
      Returns the layer builders given a string representation such as "Input(10, 0.2)|ReLU(50, 0.5)|Sigmoid(30, 0.5)|...".
      Parameters:
      k - the number of classes. k < 2 for regression.
      p - the number of input variables (not including bias value).
      spec - the hidden layer specification.
      Returns:
      the layer builders.