Class FileBatchRecordReader
- java.lang.Object
-
- org.datavec.api.records.reader.impl.filebatch.FileBatchRecordReader
-
- All Implemented Interfaces:
Closeable
,Serializable
,AutoCloseable
,Configurable
,RecordReader
public class FileBatchRecordReader extends Object implements RecordReader
- See Also:
- Serialized Form
-
-
Field Summary
-
Fields inherited from interface org.datavec.api.records.reader.RecordReader
APPEND_LABEL, LABELS, NAME_SPACE
-
-
Constructor Summary
Constructors Constructor Description FileBatchRecordReader(RecordReader rr, FileBatch fileBatch)
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description boolean
batchesSupported()
This method returns true, if next(int) signature is supported by this RecordReader implementation.void
close()
Configuration
getConf()
Return the configuration used by this object.List<String>
getLabels()
List of label stringsList<RecordListener>
getListeners()
Get the record listeners for this record reader.boolean
hasNext()
Whether there are anymore recordsvoid
initialize(Configuration conf, InputSplit split)
Called once at initialization.void
initialize(InputSplit split)
Called once at initialization.List<Record>
loadFromMetaData(List<RecordMetaData> recordMetaDatas)
Load multiple records from the given a list ofRecordMetaData
instancesRecord
loadFromMetaData(RecordMetaData recordMetaData)
Load a single record from the givenRecordMetaData
instance
Note: that for data that isn't splittable (i.e., text data that needs to be scanned/split), it is more efficient to load multiple records at once usingRecordReader.loadFromMetaData(List)
List<Writable>
next()
Get the next recordList<List<Writable>>
next(int num)
This method will be used, if batchesSupported() returns true.Record
nextRecord()
Similar toRecordReader.next()
, but returns aRecord
object, that may include metadata such as the source of the dataList<Writable>
record(URI uri, DataInputStream dataInputStream)
Load the record from the given DataInputStream UnlikeRecordReader.next()
the internal state of the RecordReader is not modified Implementations of this method should not close the DataInputStreamvoid
reset()
Reset record reader iteratorboolean
resetSupported()
void
setConf(Configuration conf)
Set the configuration to be used by this object.void
setListeners(Collection<RecordListener> listeners)
Set the record listeners for this record reader.void
setListeners(RecordListener... listeners)
Set the record listeners for this record reader.
-
-
-
Constructor Detail
-
FileBatchRecordReader
public FileBatchRecordReader(RecordReader rr, FileBatch fileBatch)
- Parameters:
rr
- Underlying record reader to read files fromfileBatch
- File batch to read files from
-
-
Method Detail
-
initialize
public void initialize(InputSplit split) throws IOException, InterruptedException
Description copied from interface:RecordReader
Called once at initialization.- Specified by:
initialize
in interfaceRecordReader
- Parameters:
split
- the split that defines the range of records to read- Throws:
IOException
InterruptedException
-
initialize
public void initialize(Configuration conf, InputSplit split) throws IOException, InterruptedException
Description copied from interface:RecordReader
Called once at initialization.- Specified by:
initialize
in interfaceRecordReader
- Parameters:
conf
- a configuration for initializationsplit
- the split that defines the range of records to read- Throws:
IOException
InterruptedException
-
batchesSupported
public boolean batchesSupported()
Description copied from interface:RecordReader
This method returns true, if next(int) signature is supported by this RecordReader implementation.- Specified by:
batchesSupported
in interfaceRecordReader
- Returns:
-
next
public List<List<Writable>> next(int num)
Description copied from interface:RecordReader
This method will be used, if batchesSupported() returns true.- Specified by:
next
in interfaceRecordReader
- Returns:
-
next
public List<Writable> next()
Description copied from interface:RecordReader
Get the next record- Specified by:
next
in interfaceRecordReader
- Returns:
-
hasNext
public boolean hasNext()
Description copied from interface:RecordReader
Whether there are anymore records- Specified by:
hasNext
in interfaceRecordReader
- Returns:
-
getLabels
public List<String> getLabels()
Description copied from interface:RecordReader
List of label strings- Specified by:
getLabels
in interfaceRecordReader
- Returns:
-
reset
public void reset()
Description copied from interface:RecordReader
Reset record reader iterator- Specified by:
reset
in interfaceRecordReader
-
resetSupported
public boolean resetSupported()
- Specified by:
resetSupported
in interfaceRecordReader
- Returns:
- True if the record reader can be reset, false otherwise. Note that some record readers cannot be reset - for example, if they are backed by a non-resettable input split (such as certain types of streams)
-
record
public List<Writable> record(URI uri, DataInputStream dataInputStream) throws IOException
Description copied from interface:RecordReader
Load the record from the given DataInputStream UnlikeRecordReader.next()
the internal state of the RecordReader is not modified Implementations of this method should not close the DataInputStream- Specified by:
record
in interfaceRecordReader
- Throws:
IOException
- if error occurs during reading from the input stream
-
nextRecord
public Record nextRecord()
Description copied from interface:RecordReader
Similar toRecordReader.next()
, but returns aRecord
object, that may include metadata such as the source of the data- Specified by:
nextRecord
in interfaceRecordReader
- Returns:
- next record
-
loadFromMetaData
public Record loadFromMetaData(RecordMetaData recordMetaData) throws IOException
Description copied from interface:RecordReader
Load a single record from the givenRecordMetaData
instance
Note: that for data that isn't splittable (i.e., text data that needs to be scanned/split), it is more efficient to load multiple records at once usingRecordReader.loadFromMetaData(List)
- Specified by:
loadFromMetaData
in interfaceRecordReader
- Parameters:
recordMetaData
- Metadata for the record that we want to load from- Returns:
- Single record for the given RecordMetaData instance
- Throws:
IOException
- If I/O error occurs during loading
-
loadFromMetaData
public List<Record> loadFromMetaData(List<RecordMetaData> recordMetaDatas) throws IOException
Description copied from interface:RecordReader
Load multiple records from the given a list ofRecordMetaData
instances- Specified by:
loadFromMetaData
in interfaceRecordReader
- Parameters:
recordMetaDatas
- Metadata for the records that we want to load from- Returns:
- Multiple records for the given RecordMetaData instances
- Throws:
IOException
- If I/O error occurs during loading
-
getListeners
public List<RecordListener> getListeners()
Description copied from interface:RecordReader
Get the record listeners for this record reader.- Specified by:
getListeners
in interfaceRecordReader
-
setListeners
public void setListeners(RecordListener... listeners)
Description copied from interface:RecordReader
Set the record listeners for this record reader.- Specified by:
setListeners
in interfaceRecordReader
-
setListeners
public void setListeners(Collection<RecordListener> listeners)
Description copied from interface:RecordReader
Set the record listeners for this record reader.- Specified by:
setListeners
in interfaceRecordReader
-
close
public void close() throws IOException
- Specified by:
close
in interfaceAutoCloseable
- Specified by:
close
in interfaceCloseable
- Throws:
IOException
-
setConf
public void setConf(Configuration conf)
Description copied from interface:Configurable
Set the configuration to be used by this object.- Specified by:
setConf
in interfaceConfigurable
-
getConf
public Configuration getConf()
Description copied from interface:Configurable
Return the configuration used by this object.- Specified by:
getConf
in interfaceConfigurable
-
-