org.deeplearning4j.optimize.solvers.accumulation

Class EncodedGradientsAccumulator

java.lang.Object

org.deeplearning4j.optimize.solvers.accumulation.EncodedGradientsAccumulator

All Implemented Interfaces:

Serializable, GradientsAccumulator, Registerable

public class EncodedGradientsAccumulator
extends Object
implements GradientsAccumulator, Registerable

This GradientsAccumulator is suited for CUDA backend.

Author:

[email protected]

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	EncodedGradientsAccumulator.Builder

Field Summary

Fields

Modifier and Type	Field and Description
protected ThreadLocal<org.nd4j.linalg.api.ndarray.INDArray>	accumulator
protected org.nd4j.linalg.api.memory.conf.WorkspaceConfiguration	appliedConfiguration
protected AtomicInteger	barrier
protected Double	boundary
protected AtomicBoolean	bypassMode
protected AtomicInteger	currentConsumers
protected Queue<org.nd4j.linalg.api.ndarray.INDArray>	externalSource
protected MessageHandler	handler
protected ThreadLocal<Integer>	index
protected long	initialMemory
protected boolean	isDebug
protected AtomicBoolean	isDone
protected AtomicBoolean	isFirst
protected List<ReentrantLock>	locks
protected List<BlockingQueue<org.nd4j.linalg.api.ndarray.INDArray>>	messages
protected int	parties
protected int	queueSize
protected AtomicBoolean	registered
protected boolean	relocatable
protected AtomicInteger	secondary
protected org.nd4j.linalg.util.AtomicThrowable	throwable
protected AtomicInteger	workersCounter
protected List<org.nd4j.linalg.api.memory.MemoryWorkspace>	workspaces

Constructor Summary

Constructors

Modifier	Constructor and Description
	EncodedGradientsAccumulator(double parties)
	EncodedGradientsAccumulator(int parties)
	EncodedGradientsAccumulator(int parties, double threshold)
protected	EncodedGradientsAccumulator(int parties, MessageHandler handler, long initialMemory, int queueSize, Double boundary)

Method Summary

Modifier and Type	Method and Description
void	applyUpdate(StepFunction function, org.nd4j.linalg.api.ndarray.INDArray params, org.nd4j.linalg.api.ndarray.INDArray updates) This method applies accumulated updates via given StepFunction
void	applyUpdate(StepFunction function, org.nd4j.linalg.api.ndarray.INDArray params, org.nd4j.linalg.api.ndarray.INDArray updates, double alpha) This method applies accumulated updates via given StepFunction
void	fallbackToSingleConsumerMode(boolean reallyFallback) This method enables/disables bypass mode
static int	getOptimalBufferSize(int paramsLength, int numWorkers, int queueSize) This method returns optimal bufferSize for a given model We know, that updates are guaranteed to have MAX size of params / 16.
static int	getOptimalBufferSize(Model model, int numWorkers, int queueSize)
void	receiveUpdate(org.nd4j.linalg.api.ndarray.INDArray array) This method accepts updates suitable for StepFunction and puts them to the queue, which is used in backpropagation loop
void	registerConsumers(int numConsumers) This method notifies producer about number of consumers for the current consumption cycle
void	reset() This method resets all accumulated updates (if any)
void	setExternalSource(Queue<org.nd4j.linalg.api.ndarray.INDArray> source) This method allows to pass external updates to accumulator, they will be populated across all workers using this GradientsAccumulator instance
void	storeUpdate(org.nd4j.linalg.api.ndarray.INDArray array) This method accepts updates suitable for StepFunction, and accumulates/propagates it across all workers
protected void	synchronize(int consumers)
protected void	synchronize(int consumers, boolean finalLock)
void	touch() This method does initialization of given worker wrt Thread-Device Affinity

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

accumulator

protected ThreadLocal<org.nd4j.linalg.api.ndarray.INDArray> accumulator

parties

protected int parties

handler

protected MessageHandler handler

messages

protected List<BlockingQueue<org.nd4j.linalg.api.ndarray.INDArray>> messages

workspaces

protected List<org.nd4j.linalg.api.memory.MemoryWorkspace> workspaces

locks

protected List<ReentrantLock> locks

workersCounter

protected AtomicInteger workersCounter

index

protected ThreadLocal<Integer> index

initialMemory

protected long initialMemory

queueSize

protected int queueSize

boundary

protected Double boundary

externalSource

protected Queue<org.nd4j.linalg.api.ndarray.INDArray> externalSource

isFirst

protected AtomicBoolean isFirst

isDone

protected AtomicBoolean isDone

barrier

protected AtomicInteger barrier

secondary

protected AtomicInteger secondary

registered

protected AtomicBoolean registered

bypassMode

protected AtomicBoolean bypassMode

currentConsumers

protected final AtomicInteger currentConsumers

throwable

protected final org.nd4j.linalg.util.AtomicThrowable throwable

isDebug

protected boolean isDebug

relocatable

protected final boolean relocatable

appliedConfiguration

protected org.nd4j.linalg.api.memory.conf.WorkspaceConfiguration appliedConfiguration

Constructor Detail

EncodedGradientsAccumulator

public EncodedGradientsAccumulator(double parties)

EncodedGradientsAccumulator

public EncodedGradientsAccumulator(int parties)

EncodedGradientsAccumulator

public EncodedGradientsAccumulator(int parties,
                                   double threshold)

EncodedGradientsAccumulator

protected EncodedGradientsAccumulator(int parties,
                                      @NonNull
                                      MessageHandler handler,
                                      long initialMemory,
                                      int queueSize,
                                      Double boundary)

Method Detail

getOptimalBufferSize

public static int getOptimalBufferSize(int paramsLength,
                                       int numWorkers,
                                       int queueSize)

This method returns optimal bufferSize for a given model We know, that updates are guaranteed to have MAX size of params / 16. So, here we go. I.e. for model with 100m params, that's 400m of floats (or 800m of doubles) The worst case for us is bitmap encoding, that takes 2 bits to encode each gradient value so, for float in worst case we'll have (100m / 16) int elements. So, our buffer size will be 6.25m * queueSize * 4 bytes per int

Parameters:

paramsLength -

numWorkers -

queueSize -

Returns:

getOptimalBufferSize

public static int getOptimalBufferSize(Model model,
                                       int numWorkers,
                                       int queueSize)

fallbackToSingleConsumerMode

public void fallbackToSingleConsumerMode(boolean reallyFallback)

Description copied from interface: Registerable

This method enables/disables bypass mode

Specified by:

fallbackToSingleConsumerMode in interface Registerable

registerConsumers

public void registerConsumers(int numConsumers)

Description copied from interface: Registerable

This method notifies producer about number of consumers for the current consumption cycle

Specified by:

registerConsumers in interface Registerable

synchronize

protected void synchronize(int consumers)

synchronize

protected void synchronize(int consumers,
                           boolean finalLock)

applyUpdate

public void applyUpdate(StepFunction function,
                        org.nd4j.linalg.api.ndarray.INDArray params,
                        org.nd4j.linalg.api.ndarray.INDArray updates)

This method applies accumulated updates via given StepFunction

Specified by:

applyUpdate in interface GradientsAccumulator

Parameters:

function -

params -

applyUpdate

public void applyUpdate(StepFunction function,
                        org.nd4j.linalg.api.ndarray.INDArray params,
                        org.nd4j.linalg.api.ndarray.INDArray updates,
                        double alpha)

This method applies accumulated updates via given StepFunction

Specified by:

applyUpdate in interface GradientsAccumulator

Parameters:

function -

params -

alpha -

setExternalSource

public void setExternalSource(Queue<org.nd4j.linalg.api.ndarray.INDArray> source)

This method allows to pass external updates to accumulator, they will be populated across all workers using this GradientsAccumulator instance

Specified by:

setExternalSource in interface GradientsAccumulator

Parameters:

source -

touch

public void touch()

This method does initialization of given worker wrt Thread-Device Affinity

Specified by:

touch in interface GradientsAccumulator

storeUpdate

public void storeUpdate(org.nd4j.linalg.api.ndarray.INDArray array)

This method accepts updates suitable for StepFunction, and accumulates/propagates it across all workers

Specified by:

storeUpdate in interface GradientsAccumulator

Parameters:

array -

receiveUpdate

public void receiveUpdate(org.nd4j.linalg.api.ndarray.INDArray array)

This method accepts updates suitable for StepFunction and puts them to the queue, which is used in backpropagation loop

PLEASE NOTE: array is expected to be ready for use and match params dimensionality

Specified by:

receiveUpdate in interface GradientsAccumulator

Parameters:

array -

reset

public void reset()

This method resets all accumulated updates (if any)

Specified by:

reset in interface GradientsAccumulator