Package org.nd4j.linalg.dataset
Class AsyncDataSetIterator
- java.lang.Object
-
- org.nd4j.linalg.dataset.AsyncDataSetIterator
-
- All Implemented Interfaces:
Serializable
,Iterator<DataSet>
,DataSetIterator
public class AsyncDataSetIterator extends Object implements DataSetIterator
- See Also:
- Serialized Form
-
-
Nested Class Summary
Nested Classes Modifier and Type Class Description protected class
AsyncDataSetIterator.AsyncPrefetchThread
-
Field Summary
Fields Modifier and Type Field Description protected DataSetIterator
backedIterator
protected BlockingQueue<DataSet>
buffer
protected DataSetCallback
callback
protected Integer
deviceId
protected AtomicBoolean
hasDepleted
protected DataSet
nextElement
protected int
prefetchSize
protected AtomicBoolean
shouldWork
protected DataSet
terminator
protected AsyncDataSetIterator.AsyncPrefetchThread
thread
protected RuntimeException
throwable
protected boolean
useWorkspace
protected String
workspaceId
-
Constructor Summary
Constructors Modifier Constructor Description protected
AsyncDataSetIterator()
AsyncDataSetIterator(DataSetIterator baseIterator)
Create an Async iterator with the default queue size of 8AsyncDataSetIterator(DataSetIterator baseIterator, int queueSize)
AsyncDataSetIterator(DataSetIterator baseIterator, int queueSize, boolean useWorkspace)
AsyncDataSetIterator(DataSetIterator baseIterator, int queueSize, boolean useWorkspace, Integer deviceId)
AsyncDataSetIterator(DataSetIterator baseIterator, int queueSize, boolean useWorkspace, DataSetCallback callback)
AsyncDataSetIterator(DataSetIterator iterator, int queueSize, BlockingQueue<DataSet> queue)
Create an Async iterator with the default queue size of 8AsyncDataSetIterator(DataSetIterator iterator, int queueSize, BlockingQueue<DataSet> queue, boolean useWorkspace)
AsyncDataSetIterator(DataSetIterator iterator, int queueSize, BlockingQueue<DataSet> queue, boolean useWorkspace, DataSetCallback callback)
AsyncDataSetIterator(DataSetIterator iterator, int queueSize, BlockingQueue<DataSet> queue, boolean useWorkspace, DataSetCallback callback, Integer deviceId)
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description boolean
asyncSupported()
Does this DataSetIterator support asynchronous prefetching of multiple DataSet objects? Most DataSetIterators do, but in some cases it may not make sense to wrap this iterator in an iterator that does asynchronous prefetching.int
batch()
Batch sizeprotected void
externalCall()
List<String>
getLabels()
Get dataset iterator record reader labelsDataSetPreProcessor
getPreProcessor()
Returns preprocessors, if definedboolean
hasNext()
Returnstrue
if the iteration has more elements.int
inputColumns()
Input columns for the datasetDataSet
next()
Returns the next element in the iteration.DataSet
next(int num)
Like the standard next method but allows a customizable number of examples returnedvoid
remove()
Removes from the underlying collection the last element returned by this iterator (optional operation).void
reset()
Resets the iterator back to the beginningboolean
resetSupported()
Is resetting supported by this DataSetIterator? Many DataSetIterators do support resetting, but some don'tvoid
setPreProcessor(DataSetPreProcessor preProcessor)
Set a pre processorvoid
shutdown()
This method will terminate background thread AND will destroy attached workspace (if any) PLEASE NOTE: After shutdown() call, this instance can't be used anymoreint
totalOutcomes()
The number of labels for the dataset-
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
-
Methods inherited from interface java.util.Iterator
forEachRemaining
-
-
-
-
Field Detail
-
backedIterator
protected DataSetIterator backedIterator
-
terminator
protected DataSet terminator
-
nextElement
protected DataSet nextElement
-
buffer
protected BlockingQueue<DataSet> buffer
-
thread
protected AsyncDataSetIterator.AsyncPrefetchThread thread
-
shouldWork
protected AtomicBoolean shouldWork
-
throwable
protected volatile RuntimeException throwable
-
useWorkspace
protected boolean useWorkspace
-
prefetchSize
protected int prefetchSize
-
workspaceId
protected String workspaceId
-
deviceId
protected Integer deviceId
-
hasDepleted
protected AtomicBoolean hasDepleted
-
callback
protected DataSetCallback callback
-
-
Constructor Detail
-
AsyncDataSetIterator
protected AsyncDataSetIterator()
-
AsyncDataSetIterator
public AsyncDataSetIterator(DataSetIterator baseIterator)
Create an Async iterator with the default queue size of 8- Parameters:
baseIterator
- Underlying iterator to wrap and fetch asynchronously from
-
AsyncDataSetIterator
public AsyncDataSetIterator(DataSetIterator iterator, int queueSize, BlockingQueue<DataSet> queue)
Create an Async iterator with the default queue size of 8- Parameters:
iterator
- Underlying iterator to wrap and fetch asynchronously fromqueue
- Queue size - number of iterators to
-
AsyncDataSetIterator
public AsyncDataSetIterator(DataSetIterator baseIterator, int queueSize)
-
AsyncDataSetIterator
public AsyncDataSetIterator(DataSetIterator baseIterator, int queueSize, boolean useWorkspace)
-
AsyncDataSetIterator
public AsyncDataSetIterator(DataSetIterator baseIterator, int queueSize, boolean useWorkspace, Integer deviceId)
-
AsyncDataSetIterator
public AsyncDataSetIterator(DataSetIterator baseIterator, int queueSize, boolean useWorkspace, DataSetCallback callback)
-
AsyncDataSetIterator
public AsyncDataSetIterator(DataSetIterator iterator, int queueSize, BlockingQueue<DataSet> queue, boolean useWorkspace)
-
AsyncDataSetIterator
public AsyncDataSetIterator(DataSetIterator iterator, int queueSize, BlockingQueue<DataSet> queue, boolean useWorkspace, DataSetCallback callback)
-
AsyncDataSetIterator
public AsyncDataSetIterator(DataSetIterator iterator, int queueSize, BlockingQueue<DataSet> queue, boolean useWorkspace, DataSetCallback callback, Integer deviceId)
-
-
Method Detail
-
next
public DataSet next(int num)
Like the standard next method but allows a customizable number of examples returned- Specified by:
next
in interfaceDataSetIterator
- Parameters:
num
- the number of examples- Returns:
- the next data applyTransformToDestination
-
inputColumns
public int inputColumns()
Input columns for the dataset- Specified by:
inputColumns
in interfaceDataSetIterator
- Returns:
-
totalOutcomes
public int totalOutcomes()
The number of labels for the dataset- Specified by:
totalOutcomes
in interfaceDataSetIterator
- Returns:
-
resetSupported
public boolean resetSupported()
Is resetting supported by this DataSetIterator? Many DataSetIterators do support resetting, but some don't- Specified by:
resetSupported
in interfaceDataSetIterator
- Returns:
- true if reset method is supported; false otherwise
-
asyncSupported
public boolean asyncSupported()
Does this DataSetIterator support asynchronous prefetching of multiple DataSet objects? Most DataSetIterators do, but in some cases it may not make sense to wrap this iterator in an iterator that does asynchronous prefetching. For example, it would not make sense to use asynchronous prefetching for the following types of iterators: (a) Iterators that store their full contents in memory already (b) Iterators that re-use features/labels arrays (as future next() calls will overwrite past contents) (c) Iterators that already implement some level of asynchronous prefetching (d) Iterators that may return different data depending on when the next() method is called- Specified by:
asyncSupported
in interfaceDataSetIterator
- Returns:
- true if asynchronous prefetching from this iterator is OK; false if asynchronous prefetching should not be used with this iterator
-
externalCall
protected void externalCall()
-
reset
public void reset()
Resets the iterator back to the beginning- Specified by:
reset
in interfaceDataSetIterator
-
shutdown
public void shutdown()
This method will terminate background thread AND will destroy attached workspace (if any) PLEASE NOTE: After shutdown() call, this instance can't be used anymore
-
batch
public int batch()
Batch size- Specified by:
batch
in interfaceDataSetIterator
- Returns:
-
setPreProcessor
public void setPreProcessor(DataSetPreProcessor preProcessor)
Set a pre processor- Specified by:
setPreProcessor
in interfaceDataSetIterator
- Parameters:
preProcessor
- a pre processor to set
-
getPreProcessor
public DataSetPreProcessor getPreProcessor()
Returns preprocessors, if defined- Specified by:
getPreProcessor
in interfaceDataSetIterator
- Returns:
-
getLabels
public List<String> getLabels()
Get dataset iterator record reader labels- Specified by:
getLabels
in interfaceDataSetIterator
-
hasNext
public boolean hasNext()
Returnstrue
if the iteration has more elements. (In other words, returnstrue
ifnext(int)
would return an element rather than throwing an exception.)
-
next
public DataSet next()
Returns the next element in the iteration.
-
remove
public void remove()
Removes from the underlying collection the last element returned by this iterator (optional operation). This method can be called only once per call tonext(int)
. The behavior of an iterator is unspecified if the underlying collection is modified while the iteration is in progress in any way other than by calling this method.- Specified by:
remove
in interfaceIterator<DataSet>
- Throws:
UnsupportedOperationException
- if theremove
operation is not supported by this iteratorIllegalStateException
- if thenext
method has not yet been called, or theremove
method has already been called after the last call to thenext
method
-
-