org.deeplearning4j.datasets.iterator

Class AsyncDataSetIterator

java.lang.Object

org.deeplearning4j.datasets.iterator.AsyncDataSetIterator

All Implemented Interfaces:

Serializable, Iterator<org.nd4j.linalg.dataset.DataSet>, org.nd4j.linalg.dataset.api.iterator.DataSetIterator

public class AsyncDataSetIterator
extends Object
implements org.nd4j.linalg.dataset.api.iterator.DataSetIterator

Async prefetching iterator wrapper for MultiDataSetIterator implementations

Author:

[email protected]

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
protected class	AsyncDataSetIterator.AsyncPrefetchThread

Field Summary

Fields

Modifier and Type	Field and Description
protected org.nd4j.linalg.dataset.api.iterator.DataSetIterator	backedIterator
protected BlockingQueue<org.nd4j.linalg.dataset.DataSet>	buffer
protected DataSetCallback	callback
protected Integer	deviceId
protected AtomicBoolean	hasDepleted
protected org.nd4j.linalg.dataset.DataSet	nextElement
protected int	prefetchSize
protected AtomicBoolean	shouldWork
protected org.nd4j.linalg.dataset.DataSet	terminator
protected AsyncDataSetIterator.AsyncPrefetchThread	thread
protected RuntimeException	throwable
protected boolean	useWorkspace
protected String	workspaceId

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	AsyncDataSetIterator()
	AsyncDataSetIterator(org.nd4j.linalg.dataset.api.iterator.DataSetIterator baseIterator)
	AsyncDataSetIterator(org.nd4j.linalg.dataset.api.iterator.DataSetIterator baseIterator, int queueSize)
	AsyncDataSetIterator(org.nd4j.linalg.dataset.api.iterator.DataSetIterator iterator, int queueSize, BlockingQueue<org.nd4j.linalg.dataset.DataSet> queue)
	AsyncDataSetIterator(org.nd4j.linalg.dataset.api.iterator.DataSetIterator iterator, int queueSize, BlockingQueue<org.nd4j.linalg.dataset.DataSet> queue, boolean useWorkspace)
	AsyncDataSetIterator(org.nd4j.linalg.dataset.api.iterator.DataSetIterator iterator, int queueSize, BlockingQueue<org.nd4j.linalg.dataset.DataSet> queue, boolean useWorkspace, DataSetCallback callback)
	AsyncDataSetIterator(org.nd4j.linalg.dataset.api.iterator.DataSetIterator iterator, int queueSize, BlockingQueue<org.nd4j.linalg.dataset.DataSet> queue, boolean useWorkspace, DataSetCallback callback, Integer deviceId)
	AsyncDataSetIterator(org.nd4j.linalg.dataset.api.iterator.DataSetIterator baseIterator, int queueSize, boolean useWorkspace)
	AsyncDataSetIterator(org.nd4j.linalg.dataset.api.iterator.DataSetIterator baseIterator, int queueSize, boolean useWorkspace, DataSetCallback callback)
	AsyncDataSetIterator(org.nd4j.linalg.dataset.api.iterator.DataSetIterator baseIterator, int queueSize, boolean useWorkspace, Integer deviceId)

Method Summary

Modifier and Type	Method and Description
boolean	asyncSupported() Does this DataSetIterator support asynchronous prefetching of multiple DataSet objects? Most DataSetIterators do, but in some cases it may not make sense to wrap this iterator in an iterator that does asynchronous prefetching.
int	batch() Batch size
int	cursor() The current cursor if applicable
protected void	externalCall()
List<String>	getLabels() Get dataset iterator record reader labels
org.nd4j.linalg.dataset.api.DataSetPreProcessor	getPreProcessor() Returns preprocessors, if defined
boolean	hasNext() Returns true if the iteration has more elements.
int	inputColumns() Input columns for the dataset
org.nd4j.linalg.dataset.DataSet	next() Returns the next element in the iteration.
org.nd4j.linalg.dataset.DataSet	next(int num) Like the standard next method but allows a customizable number of examples returned
int	numExamples() Total number of examples in the dataset
void	remove() Removes from the underlying collection the last element returned by this iterator (optional operation).
void	reset() Resets the iterator back to the beginning
boolean	resetSupported() Is resetting supported by this DataSetIterator? Many DataSetIterators do support resetting, but some don't
void	setPreProcessor(org.nd4j.linalg.dataset.api.DataSetPreProcessor preProcessor) Set a pre processor
void	shutdown() This method will terminate background thread AND will destroy attached workspace (if any) PLEASE NOTE: After shutdown() call, this instance can't be used anymore
int	totalExamples() Total examples in the iterator
int	totalOutcomes() The number of labels for the dataset

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface java.util.Iterator

forEachRemaining

Field Detail

backedIterator

protected org.nd4j.linalg.dataset.api.iterator.DataSetIterator backedIterator

terminator

protected org.nd4j.linalg.dataset.DataSet terminator

nextElement

protected org.nd4j.linalg.dataset.DataSet nextElement

buffer

protected BlockingQueue<org.nd4j.linalg.dataset.DataSet> buffer

thread

protected AsyncDataSetIterator.AsyncPrefetchThread thread

shouldWork

protected AtomicBoolean shouldWork

throwable

protected volatile RuntimeException throwable

useWorkspace

protected boolean useWorkspace

prefetchSize

protected int prefetchSize

workspaceId

protected String workspaceId

deviceId

protected Integer deviceId

hasDepleted

protected AtomicBoolean hasDepleted

callback

protected DataSetCallback callback

Constructor Detail

AsyncDataSetIterator

protected AsyncDataSetIterator()

AsyncDataSetIterator

public AsyncDataSetIterator(org.nd4j.linalg.dataset.api.iterator.DataSetIterator baseIterator)

AsyncDataSetIterator

public AsyncDataSetIterator(org.nd4j.linalg.dataset.api.iterator.DataSetIterator iterator,
                            int queueSize,
                            BlockingQueue<org.nd4j.linalg.dataset.DataSet> queue)

AsyncDataSetIterator

public AsyncDataSetIterator(org.nd4j.linalg.dataset.api.iterator.DataSetIterator baseIterator,
                            int queueSize)

AsyncDataSetIterator

public AsyncDataSetIterator(org.nd4j.linalg.dataset.api.iterator.DataSetIterator baseIterator,
                            int queueSize,
                            boolean useWorkspace)

AsyncDataSetIterator

public AsyncDataSetIterator(org.nd4j.linalg.dataset.api.iterator.DataSetIterator baseIterator,
                            int queueSize,
                            boolean useWorkspace,
                            Integer deviceId)

AsyncDataSetIterator

public AsyncDataSetIterator(org.nd4j.linalg.dataset.api.iterator.DataSetIterator baseIterator,
                            int queueSize,
                            boolean useWorkspace,
                            DataSetCallback callback)

AsyncDataSetIterator

public AsyncDataSetIterator(org.nd4j.linalg.dataset.api.iterator.DataSetIterator iterator,
                            int queueSize,
                            BlockingQueue<org.nd4j.linalg.dataset.DataSet> queue,
                            boolean useWorkspace)

AsyncDataSetIterator

public AsyncDataSetIterator(org.nd4j.linalg.dataset.api.iterator.DataSetIterator iterator,
                            int queueSize,
                            BlockingQueue<org.nd4j.linalg.dataset.DataSet> queue,
                            boolean useWorkspace,
                            DataSetCallback callback)

AsyncDataSetIterator

public AsyncDataSetIterator(org.nd4j.linalg.dataset.api.iterator.DataSetIterator iterator,
                            int queueSize,
                            BlockingQueue<org.nd4j.linalg.dataset.DataSet> queue,
                            boolean useWorkspace,
                            DataSetCallback callback,
                            Integer deviceId)

Method Detail

next

public org.nd4j.linalg.dataset.DataSet next(int num)

Like the standard next method but allows a customizable number of examples returned

Specified by:

next in interface org.nd4j.linalg.dataset.api.iterator.DataSetIterator

Parameters:

num - the number of examples

Returns:

the next data applyTransformToDestination

totalExamples

public int totalExamples()

Total examples in the iterator

Specified by:

totalExamples in interface org.nd4j.linalg.dataset.api.iterator.DataSetIterator

Returns:

inputColumns

public int inputColumns()

Input columns for the dataset

Specified by:

inputColumns in interface org.nd4j.linalg.dataset.api.iterator.DataSetIterator

Returns:

totalOutcomes

public int totalOutcomes()

The number of labels for the dataset

Specified by:

totalOutcomes in interface org.nd4j.linalg.dataset.api.iterator.DataSetIterator

Returns:

resetSupported

public boolean resetSupported()

Is resetting supported by this DataSetIterator? Many DataSetIterators do support resetting, but some don't

Specified by:

resetSupported in interface org.nd4j.linalg.dataset.api.iterator.DataSetIterator

Returns:

true if reset method is supported; false otherwise

asyncSupported

public boolean asyncSupported()

Does this DataSetIterator support asynchronous prefetching of multiple DataSet objects? Most DataSetIterators do, but in some cases it may not make sense to wrap this iterator in an iterator that does asynchronous prefetching. For example, it would not make sense to use asynchronous prefetching for the following types of iterators: (a) Iterators that store their full contents in memory already (b) Iterators that re-use features/labels arrays (as future next() calls will overwrite past contents) (c) Iterators that already implement some level of asynchronous prefetching (d) Iterators that may return different data depending on when the next() method is called

Specified by:

asyncSupported in interface org.nd4j.linalg.dataset.api.iterator.DataSetIterator

Returns:

true if asynchronous prefetching from this iterator is OK; false if asynchronous prefetching should not be used with this iterator

externalCall

protected void externalCall()

reset

public void reset()

Resets the iterator back to the beginning

Specified by:

reset in interface org.nd4j.linalg.dataset.api.iterator.DataSetIterator

shutdown

public void shutdown()

This method will terminate background thread AND will destroy attached workspace (if any) PLEASE NOTE: After shutdown() call, this instance can't be used anymore

batch

public int batch()

Batch size

Specified by:

batch in interface org.nd4j.linalg.dataset.api.iterator.DataSetIterator

Returns:

cursor

public int cursor()

The current cursor if applicable

Specified by:

cursor in interface org.nd4j.linalg.dataset.api.iterator.DataSetIterator

Returns:

numExamples

public int numExamples()

Total number of examples in the dataset

Specified by:

numExamples in interface org.nd4j.linalg.dataset.api.iterator.DataSetIterator

Returns:

setPreProcessor

public void setPreProcessor(org.nd4j.linalg.dataset.api.DataSetPreProcessor preProcessor)

Set a pre processor

Specified by:

setPreProcessor in interface org.nd4j.linalg.dataset.api.iterator.DataSetIterator

Parameters:

preProcessor - a pre processor to set

getPreProcessor

public org.nd4j.linalg.dataset.api.DataSetPreProcessor getPreProcessor()

Returns preprocessors, if defined

Specified by:

getPreProcessor in interface org.nd4j.linalg.dataset.api.iterator.DataSetIterator

Returns:

getLabels

public List<String> getLabels()

Get dataset iterator record reader labels

Specified by:

getLabels in interface org.nd4j.linalg.dataset.api.iterator.DataSetIterator

hasNext

public boolean hasNext()

Returns true if the iteration has more elements. (In other words, returns true if next(int) would return an element rather than throwing an exception.)

Specified by:

hasNext in interface Iterator<org.nd4j.linalg.dataset.DataSet>

Returns:

true if the iteration has more elements

next

public org.nd4j.linalg.dataset.DataSet next()

Returns the next element in the iteration.

Specified by:

next in interface Iterator<org.nd4j.linalg.dataset.DataSet>

Returns:

the next element in the iteration

remove

public void remove()

Removes from the underlying collection the last element returned by this iterator (optional operation). This method can be called only once per call to next(int). The behavior of an iterator is unspecified if the underlying collection is modified while the iteration is in progress in any way other than by calling this method.

Specified by:

remove in interface Iterator<org.nd4j.linalg.dataset.DataSet>

Throws:

UnsupportedOperationException - if the remove operation is not supported by this iterator

IllegalStateException - if the next method has not yet been called, or the remove method has already been called after the last call to the next method