public class ConcurrentPartitionConsumer extends AbstractPartitionConsumer
PartitionConsumer
that supports multiple instances consuming the same set of partitions
by using a working set of partitions, and keeping track of their progress state during processing
of those partitions.Constructor and Description |
---|
ConcurrentPartitionConsumer(PartitionedFileSet partitionedFileSet,
StatePersistor statePersistor) |
ConcurrentPartitionConsumer(PartitionedFileSet partitionedFileSet,
StatePersistor statePersistor,
ConsumerConfiguration configuration) |
Modifier and Type | Method and Description |
---|---|
protected void |
abort(ConsumerWorkingSet workingSet,
List<? extends PartitionKey> partitionKeys)
Resets the process state of the given partition keys, as they were not successfully processed,
or discards the partition if it has already been attempted the configured number of attempts.
|
protected void |
assertInProgress(ConsumablePartition consumablePartition)
ensure that caller doesn't try to commit/abort a partition that isn't in progress
|
protected void |
commit(ConsumerWorkingSet workingSet,
List<? extends PartitionKey> partitionKeys)
Removes the given partition keys from the working set, as they have been successfully
processed.
|
PartitionConsumerResult |
doConsume(ConsumerWorkingSet workingSet,
PartitionAcceptor acceptor)
This method will be called whenever a
PartitionConsumer requests partitions, to consume
partitions from the working set, while marking them as IN_PROGRESS |
protected void |
doExpiry(ConsumerWorkingSet workingSet)
Goes through all partitions.
|
void |
doFinish(ConsumerWorkingSet workingSet,
List<? extends PartitionKey> partitionKeys,
boolean succeeded)
This method will be called on any partitions returned by the
#consumePartitions
method. |
protected long |
getExpiryBorder() |
protected List<PartitionDetail> |
removeDiscardedPartitions(ConsumerWorkingSet workingSet)
Removes the list of partitions that have failed processing the configured number of times from
the working set and returns them.
|
void |
untake(ConsumerWorkingSet workingSet,
List<? extends PartitionKey> partitionKeys)
Returns a list of partition keys to the working set, without increasing the number of retries.
|
consumePartitions, consumePartitions, consumePartitions, getConfiguration, getPartitionedFileSet, getStatePersistor, onFinish, onFinishWithKeys, untake, untakeWithKeys
public ConcurrentPartitionConsumer(PartitionedFileSet partitionedFileSet, StatePersistor statePersistor)
public ConcurrentPartitionConsumer(PartitionedFileSet partitionedFileSet, StatePersistor statePersistor, ConsumerConfiguration configuration)
public PartitionConsumerResult doConsume(ConsumerWorkingSet workingSet, PartitionAcceptor acceptor)
AbstractPartitionConsumer
PartitionConsumer
requests partitions, to consume
partitions from the working set, while marking them as IN_PROGRESSdoConsume
in class AbstractPartitionConsumer
acceptor
- a PartitionAcceptor
which defines which partitions to
accept/consumePartitionConsumerResult
representing the consumed partitionspublic void doFinish(ConsumerWorkingSet workingSet, List<? extends PartitionKey> partitionKeys, boolean succeeded)
AbstractPartitionConsumer
#consumePartitions
method.doFinish
in class AbstractPartitionConsumer
workingSet
- the working set of partitions to operate onpartitionKeys
- list of partition keys to mark as either succeeded or failed
processingsucceeded
- whether or not processing of the specified partitions was successfulpublic void untake(ConsumerWorkingSet workingSet, List<? extends PartitionKey> partitionKeys)
AbstractPartitionConsumer
untake
in class AbstractPartitionConsumer
workingSet
- the working set of partitions to operate onpartitionKeys
- the list of partition keys to return to the working setprotected void commit(ConsumerWorkingSet workingSet, List<? extends PartitionKey> partitionKeys)
protected void abort(ConsumerWorkingSet workingSet, List<? extends PartitionKey> partitionKeys)
protected void assertInProgress(ConsumablePartition consumablePartition)
IllegalArgumentException
- if the given partition is in progressprotected List<PartitionDetail> removeDiscardedPartitions(ConsumerWorkingSet workingSet)
protected long getExpiryBorder()
protected void doExpiry(ConsumerWorkingSet workingSet)
Copyright © 2024 Cask Data, Inc. Licensed under the Apache License, Version 2.0.