Package org.datavec.api.split.partition
Class NumberOfRecordsPartitioner
- java.lang.Object
-
- org.datavec.api.split.partition.NumberOfRecordsPartitioner
-
- All Implemented Interfaces:
Partitioner
public class NumberOfRecordsPartitioner extends Object implements Partitioner
-
-
Field Summary
Fields Modifier and Type Field Description static int
DEFAULT_RECORDS_PER_FILE
static String
RECORDS_PER_FILE_CONFIG
-
Constructor Summary
Constructors Constructor Description NumberOfRecordsPartitioner()
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description OutputStream
currentOutputStream()
Get the current output streamvoid
init(Configuration configuration, InputSplit split)
Initializes this partitioner with the given configuration and input splitvoid
init(InputSplit inputSplit)
Initializes this partitioner with the given configuration and input splitboolean
needsNewPartition()
Returns true if the partition needs to be moved to the next.int
numPartitions()
Returns the number of partitionsint
numRecordsWritten()
Number of records written so farOutputStream
openNewStream()
"Increment" to the next streamint
totalRecordsWritten()
Returns the total records writtenvoid
updatePartitionInfo(PartitionMetaData metadata)
Updates the metadata for this partitioner (to indicate whether the next partition is needed or not)
-
-
-
Field Detail
-
DEFAULT_RECORDS_PER_FILE
public static final int DEFAULT_RECORDS_PER_FILE
- See Also:
- Constant Field Values
-
RECORDS_PER_FILE_CONFIG
public static final String RECORDS_PER_FILE_CONFIG
- See Also:
- Constant Field Values
-
-
Method Detail
-
totalRecordsWritten
public int totalRecordsWritten()
Description copied from interface:Partitioner
Returns the total records written- Specified by:
totalRecordsWritten
in interfacePartitioner
- Returns:
-
numRecordsWritten
public int numRecordsWritten()
Description copied from interface:Partitioner
Number of records written so far- Specified by:
numRecordsWritten
in interfacePartitioner
- Returns:
-
numPartitions
public int numPartitions()
Description copied from interface:Partitioner
Returns the number of partitions- Specified by:
numPartitions
in interfacePartitioner
- Returns:
-
init
public void init(InputSplit inputSplit)
Description copied from interface:Partitioner
Initializes this partitioner with the given configuration and input split- Specified by:
init
in interfacePartitioner
- Parameters:
inputSplit
- the input split to use with this partitioner
-
init
public void init(Configuration configuration, InputSplit split)
Description copied from interface:Partitioner
Initializes this partitioner with the given configuration and input split- Specified by:
init
in interfacePartitioner
- Parameters:
configuration
- the configuration to configure this partitioner withsplit
- the input split to use with this partitioner
-
updatePartitionInfo
public void updatePartitionInfo(PartitionMetaData metadata)
Description copied from interface:Partitioner
Updates the metadata for this partitioner (to indicate whether the next partition is needed or not)- Specified by:
updatePartitionInfo
in interfacePartitioner
-
needsNewPartition
public boolean needsNewPartition()
Description copied from interface:Partitioner
Returns true if the partition needs to be moved to the next. This is controlled withPartitioner.updatePartitionInfo(PartitionMetaData)
which handles incrementing counters and the like to determine whether the current partition has been exhausted.- Specified by:
needsNewPartition
in interfacePartitioner
- Returns:
-
openNewStream
public OutputStream openNewStream()
Description copied from interface:Partitioner
"Increment" to the next stream- Specified by:
openNewStream
in interfacePartitioner
- Returns:
- the new opened output stream
-
currentOutputStream
public OutputStream currentOutputStream()
Description copied from interface:Partitioner
Get the current output stream- Specified by:
currentOutputStream
in interfacePartitioner
- Returns:
-
-