Class InMemorySequenceRecordReader
- java.lang.Object
-
- org.datavec.api.records.reader.impl.inmemory.InMemorySequenceRecordReader
-
- All Implemented Interfaces:
Closeable
,Serializable
,AutoCloseable
,Configurable
,RecordReader
,SequenceRecordReader
public class InMemorySequenceRecordReader extends Object implements SequenceRecordReader
- See Also:
- Serialized Form
-
-
Field Summary
-
Fields inherited from interface org.datavec.api.records.reader.RecordReader
APPEND_LABEL, LABELS, NAME_SPACE
-
-
Constructor Summary
Constructors Constructor Description InMemorySequenceRecordReader(List<List<List<Writable>>> records)
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description boolean
batchesSupported()
This method returns true, if next(int) signature is supported by this RecordReader implementation.void
close()
Closes this stream and releases any system resources associated with it.Configuration
getConf()
Return the configuration used by this object.List<String>
getLabels()
List of label stringsList<RecordListener>
getListeners()
Get the record listeners for this record reader.boolean
hasNext()
Whether there are anymore recordsvoid
initialize(Configuration conf, InputSplit split)
Called once at initialization.void
initialize(InputSplit split)
Called once at initialization.List<Record>
loadFromMetaData(List<RecordMetaData> recordMetaDatas)
Load multiple records from the given a list ofRecordMetaData
instancesRecord
loadFromMetaData(RecordMetaData recordMetaData)
Load a single record from the givenRecordMetaData
instance
Note: that for data that isn't splittable (i.e., text data that needs to be scanned/split), it is more efficient to load multiple records at once usingloadFromMetaData(List)
List<SequenceRecord>
loadSequenceFromMetaData(List<RecordMetaData> recordMetaDatas)
Load multiple sequence records from the given a list ofRecordMetaData
instancesSequenceRecord
loadSequenceFromMetaData(RecordMetaData recordMetaData)
Load a single sequence record from the givenRecordMetaData
instance
Note: that for data that isn't splittable (i.e., text data that needs to be scanned/split), it is more efficient to load multiple records at once usingloadSequenceFromMetaData(List)
List<Writable>
next()
Get the next recordList<List<Writable>>
next(int num)
This method will be used, if batchesSupported() returns true.Record
nextRecord()
SequenceRecord
nextSequence()
Similar tosequenceRecord()
, but returns aRecord
object, that may include metadata such as the source of the dataList<Writable>
record(URI uri, DataInputStream dataInputStream)
Load the record from the given DataInputStream Unlikenext()
the internal state of the RecordReader is not modified Implementations of this method should not close the DataInputStreamvoid
reset()
Reset record reader iteratorboolean
resetSupported()
List<List<Writable>>
sequenceRecord()
Returns a sequence record.List<List<Writable>>
sequenceRecord(URI uri, DataInputStream dataInputStream)
Load a sequence record from the given DataInputStream Unlikenext()
the internal state of the RecordReader is not modified Implementations of this method should not close the DataInputStreamvoid
setConf(Configuration conf)
Set the configuration to be used by this object.void
setListeners(Collection<RecordListener> listeners)
Set the record listeners for this record reader.void
setListeners(RecordListener... listeners)
Set the record listeners for this record reader.
-
-
-
Method Detail
-
setConf
public void setConf(Configuration conf)
Set the configuration to be used by this object.- Specified by:
setConf
in interfaceConfigurable
- Parameters:
conf
-
-
getConf
public Configuration getConf()
Return the configuration used by this object.- Specified by:
getConf
in interfaceConfigurable
-
sequenceRecord
public List<List<Writable>> sequenceRecord()
Returns a sequence record.- Specified by:
sequenceRecord
in interfaceSequenceRecordReader
- Returns:
- a sequence of records
-
batchesSupported
public boolean batchesSupported()
Description copied from interface:RecordReader
This method returns true, if next(int) signature is supported by this RecordReader implementation.- Specified by:
batchesSupported
in interfaceRecordReader
- Returns:
-
next
public List<List<Writable>> next(int num)
Description copied from interface:RecordReader
This method will be used, if batchesSupported() returns true.- Specified by:
next
in interfaceRecordReader
- Returns:
-
sequenceRecord
public List<List<Writable>> sequenceRecord(URI uri, DataInputStream dataInputStream) throws IOException
Load a sequence record from the given DataInputStream Unlikenext()
the internal state of the RecordReader is not modified Implementations of this method should not close the DataInputStream- Specified by:
sequenceRecord
in interfaceSequenceRecordReader
- Parameters:
uri
-dataInputStream
-- Throws:
IOException
- if error occurs during reading from the input stream
-
nextSequence
public SequenceRecord nextSequence()
Similar tosequenceRecord()
, but returns aRecord
object, that may include metadata such as the source of the data- Specified by:
nextSequence
in interfaceSequenceRecordReader
- Returns:
- next sequence record
-
loadSequenceFromMetaData
public SequenceRecord loadSequenceFromMetaData(RecordMetaData recordMetaData) throws IOException
Load a single sequence record from the givenRecordMetaData
instance
Note: that for data that isn't splittable (i.e., text data that needs to be scanned/split), it is more efficient to load multiple records at once usingloadSequenceFromMetaData(List)
- Specified by:
loadSequenceFromMetaData
in interfaceSequenceRecordReader
- Parameters:
recordMetaData
- Metadata for the sequence record that we want to load from- Returns:
- Single sequence record for the given RecordMetaData instance
- Throws:
IOException
- If I/O error occurs during loading
-
loadSequenceFromMetaData
public List<SequenceRecord> loadSequenceFromMetaData(List<RecordMetaData> recordMetaDatas) throws IOException
Load multiple sequence records from the given a list ofRecordMetaData
instances- Specified by:
loadSequenceFromMetaData
in interfaceSequenceRecordReader
- Parameters:
recordMetaDatas
- Metadata for the records that we want to load from- Returns:
- Multiple sequence record for the given RecordMetaData instances
- Throws:
IOException
- If I/O error occurs during loading
-
initialize
public void initialize(InputSplit split) throws IOException, InterruptedException
Called once at initialization.- Specified by:
initialize
in interfaceRecordReader
- Parameters:
split
- the split that defines the range of records to read- Throws:
IOException
InterruptedException
-
initialize
public void initialize(Configuration conf, InputSplit split) throws IOException, InterruptedException
Called once at initialization.- Specified by:
initialize
in interfaceRecordReader
- Parameters:
conf
- a configuration for initializationsplit
- the split that defines the range of records to read- Throws:
IOException
InterruptedException
-
next
public List<Writable> next()
Get the next record- Specified by:
next
in interfaceRecordReader
- Returns:
-
hasNext
public boolean hasNext()
Whether there are anymore records- Specified by:
hasNext
in interfaceRecordReader
- Returns:
-
getLabels
public List<String> getLabels()
List of label strings- Specified by:
getLabels
in interfaceRecordReader
- Returns:
-
reset
public void reset()
Reset record reader iterator- Specified by:
reset
in interfaceRecordReader
-
resetSupported
public boolean resetSupported()
- Specified by:
resetSupported
in interfaceRecordReader
- Returns:
- True if the record reader can be reset, false otherwise. Note that some record readers cannot be reset - for example, if they are backed by a non-resettable input split (such as certain types of streams)
-
record
public List<Writable> record(URI uri, DataInputStream dataInputStream) throws IOException
Load the record from the given DataInputStream Unlikenext()
the internal state of the RecordReader is not modified Implementations of this method should not close the DataInputStream- Specified by:
record
in interfaceRecordReader
- Parameters:
uri
-dataInputStream
-- Throws:
IOException
- if error occurs during reading from the input stream
-
nextRecord
public Record nextRecord()
Similar tonext()
, but returns aRecord
object, that may include metadata such as the source of the data- Specified by:
nextRecord
in interfaceRecordReader
- Returns:
- next record
-
loadFromMetaData
public Record loadFromMetaData(RecordMetaData recordMetaData) throws IOException
Load a single record from the givenRecordMetaData
instance
Note: that for data that isn't splittable (i.e., text data that needs to be scanned/split), it is more efficient to load multiple records at once usingloadFromMetaData(List)
- Specified by:
loadFromMetaData
in interfaceRecordReader
- Parameters:
recordMetaData
- Metadata for the record that we want to load from- Returns:
- Single record for the given RecordMetaData instance
- Throws:
IOException
- If I/O error occurs during loading
-
loadFromMetaData
public List<Record> loadFromMetaData(List<RecordMetaData> recordMetaDatas) throws IOException
Load multiple records from the given a list ofRecordMetaData
instances- Specified by:
loadFromMetaData
in interfaceRecordReader
- Parameters:
recordMetaDatas
- Metadata for the records that we want to load from- Returns:
- Multiple records for the given RecordMetaData instances
- Throws:
IOException
- If I/O error occurs during loading
-
getListeners
public List<RecordListener> getListeners()
Get the record listeners for this record reader.- Specified by:
getListeners
in interfaceRecordReader
-
setListeners
public void setListeners(RecordListener... listeners)
Set the record listeners for this record reader.- Specified by:
setListeners
in interfaceRecordReader
- Parameters:
listeners
-
-
setListeners
public void setListeners(Collection<RecordListener> listeners)
Set the record listeners for this record reader.- Specified by:
setListeners
in interfaceRecordReader
- Parameters:
listeners
-
-
close
public void close() throws IOException
Closes this stream and releases any system resources associated with it. If the stream is already closed then invoking this method has no effect.As noted in
AutoCloseable.close()
, cases where the close may fail require careful attention. It is strongly advised to relinquish the underlying resources and to internally mark theCloseable
as closed, prior to throwing theIOException
.- Specified by:
close
in interfaceAutoCloseable
- Specified by:
close
in interfaceCloseable
- Throws:
IOException
- if an I/O error occurs
-
-