Class InMemoryRecordReader
- java.lang.Object
-
- org.datavec.api.records.reader.impl.inmemory.InMemoryRecordReader
-
- All Implemented Interfaces:
Closeable
,Serializable
,AutoCloseable
,Configurable
,RecordReader
public class InMemoryRecordReader extends Object implements RecordReader
- See Also:
- Serialized Form
-
-
Field Summary
-
Fields inherited from interface org.datavec.api.records.reader.RecordReader
APPEND_LABEL, LABELS, NAME_SPACE
-
-
Constructor Summary
Constructors Constructor Description InMemoryRecordReader(List<List<Writable>> records)
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description boolean
batchesSupported()
This method returns true, if next(int) signature is supported by this RecordReader implementation.void
close()
Closes this stream and releases any system resources associated with it.Configuration
getConf()
Return the configuration used by this object.List<String>
getLabels()
List of label stringsList<RecordListener>
getListeners()
Get the record listeners for this record reader.boolean
hasNext()
Whether there are anymore recordsvoid
initialize(Configuration conf, InputSplit split)
Called once at initialization.void
initialize(InputSplit split)
Called once at initialization.List<Record>
loadFromMetaData(List<RecordMetaData> recordMetaDatas)
Load multiple records from the given a list ofRecordMetaData
instancesRecord
loadFromMetaData(RecordMetaData recordMetaData)
Load a single record from the givenRecordMetaData
instance
Note: that for data that isn't splittable (i.e., text data that needs to be scanned/split), it is more efficient to load multiple records at once usingloadFromMetaData(List)
List<Writable>
next()
Get the next recordList<List<Writable>>
next(int num)
This method will be used, if batchesSupported() returns true.Record
nextRecord()
List<Writable>
record(URI uri, DataInputStream dataInputStream)
Load the record from the given DataInputStream Unlikenext()
the internal state of the RecordReader is not modified Implementations of this method should not close the DataInputStreamvoid
reset()
Reset record reader iteratorboolean
resetSupported()
void
setConf(Configuration conf)
Set the configuration to be used by this object.void
setListeners(Collection<RecordListener> listeners)
Set the record listeners for this record reader.void
setListeners(RecordListener... listeners)
Set the record listeners for this record reader.
-
-
-
Method Detail
-
initialize
public void initialize(InputSplit split) throws IOException, InterruptedException
Called once at initialization.- Specified by:
initialize
in interfaceRecordReader
- Parameters:
split
- the split that defines the range of records to read- Throws:
IOException
InterruptedException
-
initialize
public void initialize(Configuration conf, InputSplit split) throws IOException, InterruptedException
Called once at initialization.- Specified by:
initialize
in interfaceRecordReader
- Parameters:
conf
- a configuration for initializationsplit
- the split that defines the range of records to read- Throws:
IOException
InterruptedException
-
batchesSupported
public boolean batchesSupported()
Description copied from interface:RecordReader
This method returns true, if next(int) signature is supported by this RecordReader implementation.- Specified by:
batchesSupported
in interfaceRecordReader
- Returns:
-
next
public List<List<Writable>> next(int num)
Description copied from interface:RecordReader
This method will be used, if batchesSupported() returns true.- Specified by:
next
in interfaceRecordReader
- Returns:
-
next
public List<Writable> next()
Get the next record- Specified by:
next
in interfaceRecordReader
- Returns:
-
hasNext
public boolean hasNext()
Whether there are anymore records- Specified by:
hasNext
in interfaceRecordReader
- Returns:
-
getLabels
public List<String> getLabels()
List of label strings- Specified by:
getLabels
in interfaceRecordReader
- Returns:
-
reset
public void reset()
Reset record reader iterator- Specified by:
reset
in interfaceRecordReader
-
resetSupported
public boolean resetSupported()
- Specified by:
resetSupported
in interfaceRecordReader
- Returns:
- True if the record reader can be reset, false otherwise. Note that some record readers cannot be reset - for example, if they are backed by a non-resettable input split (such as certain types of streams)
-
record
public List<Writable> record(URI uri, DataInputStream dataInputStream) throws IOException
Load the record from the given DataInputStream Unlikenext()
the internal state of the RecordReader is not modified Implementations of this method should not close the DataInputStream- Specified by:
record
in interfaceRecordReader
- Parameters:
uri
-dataInputStream
-- Throws:
IOException
- if error occurs during reading from the input stream
-
nextRecord
public Record nextRecord()
Similar tonext()
, but returns aRecord
object, that may include metadata such as the source of the data- Specified by:
nextRecord
in interfaceRecordReader
- Returns:
- next record
-
loadFromMetaData
public Record loadFromMetaData(RecordMetaData recordMetaData) throws IOException
Load a single record from the givenRecordMetaData
instance
Note: that for data that isn't splittable (i.e., text data that needs to be scanned/split), it is more efficient to load multiple records at once usingloadFromMetaData(List)
- Specified by:
loadFromMetaData
in interfaceRecordReader
- Parameters:
recordMetaData
- Metadata for the record that we want to load from- Returns:
- Single record for the given RecordMetaData instance
- Throws:
IOException
- If I/O error occurs during loading
-
loadFromMetaData
public List<Record> loadFromMetaData(List<RecordMetaData> recordMetaDatas) throws IOException
Load multiple records from the given a list ofRecordMetaData
instances- Specified by:
loadFromMetaData
in interfaceRecordReader
- Parameters:
recordMetaDatas
- Metadata for the records that we want to load from- Returns:
- Multiple records for the given RecordMetaData instances
- Throws:
IOException
- If I/O error occurs during loading
-
getListeners
public List<RecordListener> getListeners()
Get the record listeners for this record reader.- Specified by:
getListeners
in interfaceRecordReader
-
setListeners
public void setListeners(RecordListener... listeners)
Set the record listeners for this record reader.- Specified by:
setListeners
in interfaceRecordReader
- Parameters:
listeners
-
-
setListeners
public void setListeners(Collection<RecordListener> listeners)
Set the record listeners for this record reader.- Specified by:
setListeners
in interfaceRecordReader
- Parameters:
listeners
-
-
close
public void close() throws IOException
Closes this stream and releases any system resources associated with it. If the stream is already closed then invoking this method has no effect.As noted in
AutoCloseable.close()
, cases where the close may fail require careful attention. It is strongly advised to relinquish the underlying resources and to internally mark theCloseable
as closed, prior to throwing theIOException
.- Specified by:
close
in interfaceAutoCloseable
- Specified by:
close
in interfaceCloseable
- Throws:
IOException
- if an I/O error occurs
-
setConf
public void setConf(Configuration conf)
Set the configuration to be used by this object.- Specified by:
setConf
in interfaceConfigurable
- Parameters:
conf
-
-
getConf
public Configuration getConf()
Return the configuration used by this object.- Specified by:
getConf
in interfaceConfigurable
-
-