@Beta public class PartitionBatchInput extends Object
PartitionedFileSet
as input to a MapReduceContext with runtime arguments to
appropriately process partitions.Modifier and Type | Class and Description |
---|---|
static interface |
PartitionBatchInput.BatchPartitionCommitter
Used to persist the state of the partition consumer.
|
Constructor and Description |
---|
PartitionBatchInput() |
Modifier and Type | Method and Description |
---|---|
static PartitionBatchInput.BatchPartitionCommitter |
setInput(MapReduceContext mapreduceContext,
String partitionedFileSetName,
DatasetStatePersistor statePersistor)
|
static PartitionBatchInput.BatchPartitionCommitter |
setInput(MapReduceContext mapreduceContext,
String partitionedFileSetName,
DatasetStatePersistor statePersistor,
ConsumerConfiguration consumerConfiguration)
Used from the initialize method of the implementing batch job to configure as input a
PartitionedFileSet that has specified a set of
Partition s of a PartitionedFileSet to be processed by the run of the batch job. |
public static PartitionBatchInput.BatchPartitionCommitter setInput(MapReduceContext mapreduceContext, String partitionedFileSetName, DatasetStatePersistor statePersistor)
public static PartitionBatchInput.BatchPartitionCommitter setInput(MapReduceContext mapreduceContext, String partitionedFileSetName, DatasetStatePersistor statePersistor, ConsumerConfiguration consumerConfiguration)
Partition
s of a PartitionedFileSet
to be processed by the run of the batch job. It does this by reading back
the previous state, determining the new partitions to read, computing the new state, and
persisting this new state. It then configures this dataset as input to the mapreduce context
that is passed in.mapreduceContext
- MapReduce context used to access the PartitionedFileSet, and on
which the input is configuredpartitionedFileSetName
- the name of the PartitionedFileSet
to consume
partitions fromstatePersistor
- a DatasetStatePersistor
responsible for defining how the
partition consumer state is managedconsumerConfiguration
- defines parameters for the partition consumptionCopyright © 2024 Cask Data, Inc. Licensed under the Apache License, Version 2.0.