org.deeplearning4j.nn

Class BaseNeuralNetwork

java.lang.Object

org.deeplearning4j.nn.BaseNeuralNetwork

All Implemented Interfaces:

Serializable, Cloneable, NeuralNetwork, Persistable, NeuralNetEpochListener

Direct Known Subclasses:

DenoisingAutoEncoder, RBM

public abstract class BaseNeuralNetwork
extends Object
implements NeuralNetwork, Persistable

Baseline class for any Neural Network used as a layer in a deep network such as an DBN

Author:

Adam Gibson

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	BaseNeuralNetwork.Builder<E extends BaseNeuralNetwork>

Nested classes/interfaces inherited from interface org.deeplearning4j.nn.NeuralNetwork

NeuralNetwork.LossFunction, NeuralNetwork.OptimizationAlgorithm

Field Summary

Fields

Modifier and Type	Field and Description
protected boolean	applySparsity
protected org.apache.commons.math3.distribution.RealDistribution	dist
protected org.jblas.DoubleMatrix	doMask
protected double	dropOut
protected double	fanIn
protected boolean	firstTimeThrough
protected List<NeuralNetworkGradientListener>	gradientListeners
protected org.jblas.DoubleMatrix	hBias
protected AdaGrad	hBiasAdaGrad
protected org.jblas.DoubleMatrix	input
protected double	l2
protected NeuralNetwork.LossFunction	lossFunction
protected double	momentum
protected int	nHidden Number of hidden units One tip with this is usually having more hidden units than inputs (read: input rows here) will typically cause terrible overfitting.
protected boolean	normalizeByInputRows
int	nVisible
protected NeuralNetwork.OptimizationAlgorithm	optimizationAlgo
protected NeuralNetworkOptimizer	optimizer
protected int	renderWeightsEveryNumEpochs
protected org.apache.commons.math3.random.RandomGenerator	rng
protected double	sparsity
protected boolean	useAdaGrad
protected boolean	useRegularization
protected org.jblas.DoubleMatrix	vBias
protected AdaGrad	vBiasAdaGrad
protected org.jblas.DoubleMatrix	W
protected AdaGrad	wAdaGrad

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	BaseNeuralNetwork()
	BaseNeuralNetwork(org.jblas.DoubleMatrix input, int nVisible, int nHidden, org.jblas.DoubleMatrix W, org.jblas.DoubleMatrix hbias, org.jblas.DoubleMatrix vbias, org.apache.commons.math3.random.RandomGenerator rng, double fanIn, org.apache.commons.math3.distribution.RealDistribution dist)
	BaseNeuralNetwork(int nVisible, int nHidden, org.jblas.DoubleMatrix W, org.jblas.DoubleMatrix hbias, org.jblas.DoubleMatrix vbias, org.apache.commons.math3.random.RandomGenerator rng, double fanIn, org.apache.commons.math3.distribution.RealDistribution dist)

Method Summary

Methods

Modifier and Type	Method and Description
protected void	applyDropOutIfNecessary(org.jblas.DoubleMatrix input)
protected void	applySparsity(org.jblas.DoubleMatrix hBiasGradient, double learningRate) Applies sparsity to the passed in hbias gradient
NeuralNetwork	clone()
double	dropOut()
void	epochDone(int epoch) Event listener for each iteration
double	fanIn()
AdaGrad	getAdaGrad()
org.apache.commons.math3.distribution.RealDistribution	getDist()
List<NeuralNetworkGradientListener>	getGradientListeners()
org.jblas.DoubleMatrix	gethBias()
AdaGrad	gethBiasAdaGrad()
org.jblas.DoubleMatrix	getInput()
double	getL2()
NeuralNetwork.LossFunction	getLossFunction()
double	getMomentum()
int	getnHidden()
int	getnVisible()
NeuralNetwork.OptimizationAlgorithm	getOptimizationAlgorithm()
double	getReConstructionCrossEntropy() Reconstruction entropy.
int	getRenderEpochs()
org.apache.commons.math3.random.RandomGenerator	getRng()
double	getSparsity()
org.jblas.DoubleMatrix	getvBias()
AdaGrad	getVBiasAdaGrad()
org.jblas.DoubleMatrix	getW()
org.jblas.DoubleMatrix	hBiasMean()
protected void	initWeights() Initialize weights.
void	jostleWeighMatrix()
double	l2RegularizedCoefficient()
void	load(InputStream is) Load (using ObjectInputStream
double	lossFunction()
abstract double	lossFunction(Object[] params) The loss function (cross entropy, reconstruction error,...)
void	merge(NeuralNetwork network, int batchSize) Performs a network merge in the form of a += b - a / n where a is a matrix here b is a matrix on the incoming network and n is the batch size
double	negativeLogLikelihood() Negative log likelihood of the current input given the corruption level
double	negativeLoglikelihood(org.jblas.DoubleMatrix input) Negative log likelihood of the current input given the corruption level
boolean	normalizeByInputRows()
abstract org.jblas.DoubleMatrix	reconstruct(org.jblas.DoubleMatrix x) All neural networks are based on this idea of minimizing reconstruction error.
void	resetAdaGrad(double lr)
void	setAdaGrad(AdaGrad adaGrad)
void	setDist(org.apache.commons.math3.distribution.RealDistribution dist)
void	setDropOut(double dropOut)
void	setFanIn(double fanIn)
void	setGradientListeners(List<NeuralNetworkGradientListener> gradientListeners)
void	sethBias(org.jblas.DoubleMatrix hBias)
void	setHbiasAdaGrad(AdaGrad adaGrad)
void	setInput(org.jblas.DoubleMatrix input)
void	setL2(double l2)
void	setLossFunction(NeuralNetwork.LossFunction lossFunction)
void	setMomentum(double momentum)
void	setnHidden(int nHidden)
void	setnVisible(int nVisible)
void	setOptimizationAlgorithm(NeuralNetwork.OptimizationAlgorithm optimizationAlgorithm)
void	setRenderEpochs(int renderEpochs)
void	setRng(org.apache.commons.math3.random.RandomGenerator rng)
void	setSparsity(double sparsity)
void	setvBias(org.jblas.DoubleMatrix vBias)
void	setVBiasAdaGrad(AdaGrad adaGrad)
void	setW(org.jblas.DoubleMatrix w)
double	squaredLoss()
abstract void	train(org.jblas.DoubleMatrix input, double lr, Object[] params) Train one iteration of the network
NeuralNetwork	transpose()
protected void	triggerGradientEvents(NeuralNetworkGradient gradient) Triggers network gradient listeners.
void	update(BaseNeuralNetwork n) Copies params from the passed in network to this one
protected void	updateGradientAccordingToParams(NeuralNetworkGradient gradient, double learningRate) Update the gradient according to the configuration such as adagrad, momentum, and sparsity
void	write(OutputStream os) Write this to an object output stream

Methods inherited from class java.lang.Object

equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.deeplearning4j.nn.NeuralNetwork

getGradient, sampleHiddenGivenVisible, sampleVisibleGivenHidden, trainTillConvergence

Field Detail

nVisible

public int nVisible

nHidden

protected int nHidden

Number of hidden units One tip with this is usually having more hidden units than inputs (read: input rows here) will typically cause terrible overfitting. Another rule worthy of note: more training data typically results in more redundant data. It is usually a better idea to use a smaller number of hidden units.

W

protected org.jblas.DoubleMatrix W

hBias

protected org.jblas.DoubleMatrix hBias

vBias

protected org.jblas.DoubleMatrix vBias

rng

protected org.apache.commons.math3.random.RandomGenerator rng

input

protected org.jblas.DoubleMatrix input

sparsity

protected double sparsity

momentum

protected double momentum

dist

protected transient org.apache.commons.math3.distribution.RealDistribution dist

l2

protected double l2

optimizer

protected transient NeuralNetworkOptimizer optimizer

renderWeightsEveryNumEpochs

protected int renderWeightsEveryNumEpochs

fanIn

protected double fanIn

useRegularization

protected boolean useRegularization

useAdaGrad

protected boolean useAdaGrad

firstTimeThrough

protected boolean firstTimeThrough

normalizeByInputRows

protected boolean normalizeByInputRows

applySparsity

protected boolean applySparsity

gradientListeners

protected List<NeuralNetworkGradientListener> gradientListeners

dropOut

protected double dropOut

doMask

protected org.jblas.DoubleMatrix doMask

optimizationAlgo

protected NeuralNetwork.OptimizationAlgorithm optimizationAlgo

lossFunction

protected NeuralNetwork.LossFunction lossFunction

wAdaGrad

protected AdaGrad wAdaGrad

hBiasAdaGrad

protected AdaGrad hBiasAdaGrad

vBiasAdaGrad

protected AdaGrad vBiasAdaGrad

Constructor Detail

BaseNeuralNetwork

protected BaseNeuralNetwork()

BaseNeuralNetwork

public BaseNeuralNetwork(int nVisible,
                 int nHidden,
                 org.jblas.DoubleMatrix W,
                 org.jblas.DoubleMatrix hbias,
                 org.jblas.DoubleMatrix vbias,
                 org.apache.commons.math3.random.RandomGenerator rng,
                 double fanIn,
                 org.apache.commons.math3.distribution.RealDistribution dist)

Parameters:

nVisible - the number of outbound nodes

nHidden - the number of nodes in the hidden layer

W - the weights for this vector, maybe null, if so this will create a matrix with nHidden x nVisible dimensions.

hBias - the hidden bias

vBias - the visible bias (usually b for the output layer)

rng - the rng, if not a seed of 1234 is used.

BaseNeuralNetwork

public BaseNeuralNetwork(org.jblas.DoubleMatrix input,
                 int nVisible,
                 int nHidden,
                 org.jblas.DoubleMatrix W,
                 org.jblas.DoubleMatrix hbias,
                 org.jblas.DoubleMatrix vbias,
                 org.apache.commons.math3.random.RandomGenerator rng,
                 double fanIn,
                 org.apache.commons.math3.distribution.RealDistribution dist)

Parameters:

input - the input examples

nVisible - the number of outbound nodes

nHidden - the number of nodes in the hidden layer

W - the weights for this vector, maybe null, if so this will create a matrix with nHidden x nVisible dimensions.

hBias - the hidden bias

vBias - the visible bias (usually b for the output layer)

rng - the rng, if not a seed of 1234 is used.

Method Detail

l2RegularizedCoefficient

public double l2RegularizedCoefficient()

Specified by:

l2RegularizedCoefficient in interface NeuralNetwork

initWeights

protected void initWeights()

Initialize weights. This includes steps for doing a random initialization of W as well as the vbias and hbias

resetAdaGrad

public void resetAdaGrad(double lr)

Specified by:

resetAdaGrad in interface NeuralNetwork

getGradientListeners

public List<NeuralNetworkGradientListener> getGradientListeners()

Specified by:

getGradientListeners in interface NeuralNetwork

setGradientListeners

public void setGradientListeners(List<NeuralNetworkGradientListener> gradientListeners)

Specified by:

setGradientListeners in interface NeuralNetwork

setRenderEpochs

public void setRenderEpochs(int renderEpochs)

Specified by:

setRenderEpochs in interface NeuralNetwork

getRenderEpochs

public int getRenderEpochs()

Specified by:

getRenderEpochs in interface NeuralNetwork

fanIn

public double fanIn()

Specified by:

fanIn in interface NeuralNetwork

setFanIn

public void setFanIn(double fanIn)

Specified by:

setFanIn in interface NeuralNetwork

jostleWeighMatrix

public void jostleWeighMatrix()

applySparsity

protected void applySparsity(org.jblas.DoubleMatrix hBiasGradient,
                 double learningRate)

Applies sparsity to the passed in hbias gradient

Parameters:

hBiasGradient - the hbias gradient to apply to

learningRate - the learning rate used

updateGradientAccordingToParams

protected void updateGradientAccordingToParams(NeuralNetworkGradient gradient,
                                   double learningRate)

Update the gradient according to the configuration such as adagrad, momentum, and sparsity

Parameters:

gradient - the gradient to modify

learningRate - the learning rate for the current iteratiaon

triggerGradientEvents

protected void triggerGradientEvents(NeuralNetworkGradient gradient)

Triggers network gradient listeners. This should be called in getGradient

Parameters:

gradient - the gradient that triggered the event

setDropOut

public void setDropOut(double dropOut)

Specified by:

setDropOut in interface NeuralNetwork

dropOut

public double dropOut()

Specified by:

dropOut in interface NeuralNetwork

getAdaGrad

public AdaGrad getAdaGrad()

Specified by:

getAdaGrad in interface NeuralNetwork

setAdaGrad

public void setAdaGrad(AdaGrad adaGrad)

Specified by:

setAdaGrad in interface NeuralNetwork

transpose

public NeuralNetwork transpose()

Specified by:

transpose in interface NeuralNetwork

clone

public NeuralNetwork clone()

Specified by:

clone in interface NeuralNetwork

Overrides:

clone in class Object

getLossFunction

public NeuralNetwork.LossFunction getLossFunction()

Specified by:

getLossFunction in interface NeuralNetwork

setLossFunction

public void setLossFunction(NeuralNetwork.LossFunction lossFunction)

Specified by:

setLossFunction in interface NeuralNetwork

getOptimizationAlgorithm

public NeuralNetwork.OptimizationAlgorithm getOptimizationAlgorithm()

Specified by:

getOptimizationAlgorithm in interface NeuralNetwork

setOptimizationAlgorithm

public void setOptimizationAlgorithm(NeuralNetwork.OptimizationAlgorithm optimizationAlgorithm)

Specified by:

setOptimizationAlgorithm in interface NeuralNetwork

getDist

public org.apache.commons.math3.distribution.RealDistribution getDist()

Specified by:

getDist in interface NeuralNetwork

setDist

public void setDist(org.apache.commons.math3.distribution.RealDistribution dist)

Specified by:

setDist in interface NeuralNetwork

merge

public void merge(NeuralNetwork network,
         int batchSize)

Description copied from interface: NeuralNetwork

Performs a network merge in the form of a += b - a / n where a is a matrix here b is a matrix on the incoming network and n is the batch size

Specified by:

merge in interface NeuralNetwork

Parameters:

network - the network to merge with

batchSize - the batch size (number of training examples) to average by

update

public void update(BaseNeuralNetwork n)

Copies params from the passed in network to this one

Parameters:

n - the network to copy

load

public void load(InputStream is)

Load (using ObjectInputStream

Specified by:

load in interface Persistable

Parameters:

is - the input stream to load from (usually a file)

negativeLogLikelihood

public double negativeLogLikelihood()

Negative log likelihood of the current input given the corruption level

Specified by:

negativeLogLikelihood in interface NeuralNetwork

Parameters:

corruptionLevel - the corruption level to use

Returns:

the negative log likelihood of the auto encoder given the corruption level

negativeLoglikelihood

public double negativeLoglikelihood(org.jblas.DoubleMatrix input)

Negative log likelihood of the current input given the corruption level

Returns:

the negative log likelihood of the auto encoder given the corruption level

getReConstructionCrossEntropy

public double getReConstructionCrossEntropy()

Reconstruction entropy. This compares the similarity of two probability distributions, in this case that would be the input and the reconstructed input with gaussian noise. This will account for either regularization or none depending on the configuration.

Specified by:

getReConstructionCrossEntropy in interface NeuralNetwork

Returns:

reconstruction error

normalizeByInputRows

public boolean normalizeByInputRows()

Specified by:

normalizeByInputRows in interface NeuralNetwork

getnVisible

public int getnVisible()

Specified by:

getnVisible in interface NeuralNetwork

setnVisible

public void setnVisible(int nVisible)

Specified by:

setnVisible in interface NeuralNetwork

getnHidden

public int getnHidden()

Specified by:

getnHidden in interface NeuralNetwork

setnHidden

public void setnHidden(int nHidden)

Specified by:

setnHidden in interface NeuralNetwork

getW

public org.jblas.DoubleMatrix getW()

Specified by:

getW in interface NeuralNetwork

setW

public void setW(org.jblas.DoubleMatrix w)

Specified by:

setW in interface NeuralNetwork

gethBias

public org.jblas.DoubleMatrix gethBias()

Specified by:

gethBias in interface NeuralNetwork

sethBias

public void sethBias(org.jblas.DoubleMatrix hBias)

Specified by:

sethBias in interface NeuralNetwork

getvBias

public org.jblas.DoubleMatrix getvBias()

Specified by:

getvBias in interface NeuralNetwork

setvBias

public void setvBias(org.jblas.DoubleMatrix vBias)

Specified by:

setvBias in interface NeuralNetwork

getRng

public org.apache.commons.math3.random.RandomGenerator getRng()

Specified by:

getRng in interface NeuralNetwork

setRng

public void setRng(org.apache.commons.math3.random.RandomGenerator rng)

Specified by:

setRng in interface NeuralNetwork

getInput

public org.jblas.DoubleMatrix getInput()

Specified by:

getInput in interface NeuralNetwork

setInput

public void setInput(org.jblas.DoubleMatrix input)

Specified by:

setInput in interface NeuralNetwork

getSparsity

public double getSparsity()

Specified by:

getSparsity in interface NeuralNetwork

setSparsity

public void setSparsity(double sparsity)

Specified by:

setSparsity in interface NeuralNetwork

getMomentum

public double getMomentum()

Specified by:

getMomentum in interface NeuralNetwork

setMomentum

public void setMomentum(double momentum)

Specified by:

setMomentum in interface NeuralNetwork

getL2

public double getL2()

Specified by:

getL2 in interface NeuralNetwork

setL2

public void setL2(double l2)

Specified by:

setL2 in interface NeuralNetwork

gethBiasAdaGrad

public AdaGrad gethBiasAdaGrad()

Specified by:

gethBiasAdaGrad in interface NeuralNetwork

setHbiasAdaGrad

public void setHbiasAdaGrad(AdaGrad adaGrad)

Specified by:

setHbiasAdaGrad in interface NeuralNetwork

getVBiasAdaGrad

public AdaGrad getVBiasAdaGrad()

Specified by:

getVBiasAdaGrad in interface NeuralNetwork

setVBiasAdaGrad

public void setVBiasAdaGrad(AdaGrad adaGrad)

Specified by:

setVBiasAdaGrad in interface NeuralNetwork

write

public void write(OutputStream os)

Write this to an object output stream

Specified by:

write in interface Persistable

Parameters:

os - the output stream to write to

reconstruct

public abstract org.jblas.DoubleMatrix reconstruct(org.jblas.DoubleMatrix x)

All neural networks are based on this idea of minimizing reconstruction error. Both RBMs and Denoising AutoEncoders have a component for reconstructing, ala different implementations.

Parameters:

x - the input to reconstruct

Returns:

the reconstructed input

lossFunction

public abstract double lossFunction(Object[] params)

The loss function (cross entropy, reconstruction error,...)

Returns:

the loss function

lossFunction

public double lossFunction()

applyDropOutIfNecessary

protected void applyDropOutIfNecessary(org.jblas.DoubleMatrix input)

train

public abstract void train(org.jblas.DoubleMatrix input,
         double lr,
         Object[] params)

Train one iteration of the network

Specified by:

train in interface NeuralNetwork

Parameters:

input - the input to train on

lr - the learning rate to train at

params - the extra params (k, corruption level,...)

squaredLoss

public double squaredLoss()

Specified by:

squaredLoss in interface NeuralNetwork

hBiasMean

public org.jblas.DoubleMatrix hBiasMean()

Specified by:

hBiasMean in interface NeuralNetwork

epochDone

public void epochDone(int epoch)

Description copied from interface: NeuralNetEpochListener

Event listener for each iteration

Specified by:

epochDone in interface NeuralNetwork

Specified by:

epochDone in interface NeuralNetEpochListener