@Beta public interface PartitionConsumer
Partition
s of a PartitionedFileSet
. In
order to support multiple partition consumers consuming different partitions from the same
PartitionedFileSet, the consumePartitions method must be called in its own, short transaction
before the processing of the partitions. This is so that other concurrent consumers can see that
the partitions have been marked as IN_PROGRESS.Modifier and Type | Method and Description |
---|---|
PartitionConsumerResult |
consumePartitions() |
PartitionConsumerResult |
consumePartitions(int limit) |
PartitionConsumerResult |
consumePartitions(PartitionAcceptor acceptor) |
void |
onFinish(List<? extends Partition> partitions,
boolean succeeded)
This method must be called on any partitions returned by the
#consumePartitions method. |
void |
onFinishWithKeys(List<? extends PartitionKey> partitionKeys,
boolean succeeded)
|
void |
untake(List<? extends Partition> partitions)
Returns a list of partitions to the working set, without increasing the number of retries.
|
void |
untakeWithKeys(List<? extends PartitionKey> partitionKeys)
Returns a list of partition keys to the working set, without increasing the number of retries.
|
PartitionConsumerResult consumePartitions()
PartitionConsumerResult
s containing partitions that have not yet processed.PartitionConsumerResult consumePartitions(int limit)
limit
- upper limit on number of partitions to consumePartitionConsumerResult
s containing partitions that have not yet processed.PartitionConsumerResult consumePartitions(PartitionAcceptor acceptor)
acceptor
- defines which and how many partitions to consumePartitionConsumerResult
s containing partitions that have not yet processed.void onFinish(List<? extends Partition> partitions, boolean succeeded)
#consumePartitions
method.
If a program fails to call this method for any partitions, those partitions will be 'expired'
after a timeout defined on the configured ConsumerConfiguration
.partitions
- list of partitions to mark as either succeeded or failed processingsucceeded
- whether or not processing of the specified partitions was successfulIllegalStateException
- if any of the specified partitions are not in the working set
as in progress.void onFinishWithKeys(List<? extends PartitionKey> partitionKeys, boolean succeeded)
partitionKeys
- list of partition keys to mark as either succeeded or failed
processingsucceeded
- whether or not processing of the specified partitions was successfulIllegalStateException
- if any of the specified partitions are not in the working set
as in progress.void untake(List<? extends Partition> partitions)
partitions
- list of partitions to put backIllegalStateException
- if any of the specified partitions are not in the working set
as in progress.void untakeWithKeys(List<? extends PartitionKey> partitionKeys)
partitionKeys
- list of partition keys to put backIllegalStateException
- if any of the specified partitions are not in the working set
as in progress.Copyright © 2024 Cask Data, Inc. Licensed under the Apache License, Version 2.0.