flushing

Type Members

1. trait AsyncFlushing[K1, V1, K2, V2] extends Flushing with OnInit with OnClose

   

   The AsyncFlushing trait allows outstanding futures to be tracked to completion when the flush() method is called

2. abstract class AsyncProcessor[K, V] extends AbstractProcessor[K, V] with FlushingProcessor[K, V] with AsyncFlushing[K, V, Unit, Unit]

   

3. abstract class AsyncTransformer[K1, V1, K2, V2] extends FlushingTransformer[K1, V1, K2, V2] with AsyncFlushing[K1, V1, K2, V2] with ProcessorContextLogging

   

   The AsyncTransformer trait allows async futures to be used to emit records downstreams.

   The AsyncTransformer trait allows async futures to be used to emit records downstreams.

   Note

   throwIfAsyncFailure is used to fail the Kafka Streams service more quickly than waiting for an eventual failure to occur at the next commit interval. We try to fail fast and a future failure will result in your entire instance shutting down. This default behavior prevents data loss. If you want your service to handle failed futures please use handle/transform on your returned future.

   ,

   Completed Futures add output records to an outstandingResults set. Future's do not directly call context.forward on success since the Processor/Transformer classes have a defined lifecycle which revolves around 2 main processing methods (process/punctuate or transform/punctuate). Kafka Streams then ensures that these methods are never called from 2 threads at the same time. Kafka Streams assumes "forward" would only ever be called from the thread that calls process/transform/punctuate. As such, it could be dangerous to have a Finagle thread calling forward at any time. In addition, ProcessorRecordContext associated with each input record is stored in memory and reattached to the output record before flushing. This ensures fields such as headers are preserved.

   See also

   https://issues.apache.org/jira/browse/KAFKA-7432

   https://stackoverflow.com/questions/42064430/external-system-queries-during-kafka-stream-processing

   https://stackoverflow.com/questions/42049047/how-to-handle-error-and-dont-commit-when-use-kafka-streams-dsl/42056286#comment92197161_42056286

   https://issues.apache.org/jira/browse/KAFKA-6989 for related ticket in Kafka Streams backlog

4. trait Flushing extends OnInit with OnClose with OnFlush with ProcessorContextLogging

   

5. trait FlushingAwareServer extends KafkaStreamsTwitterServer

   

   FlushingAwareServer must be mixed in to servers that rely on manually controlling when a flush/commit occurs.

   FlushingAwareServer must be mixed in to servers that rely on manually controlling when a flush/commit occurs. As such, this trait will be needed when using the following classes, FlushingProcessor, FlushingTransformer, AsyncProcessor, AsyncTransformer, and FinatraTransformer

   This trait sets 'kafka.commit.interval' to 'Duration.Top' to disable the normal Kafka Streams commit process. As such the only commits that will occur are triggered manually, thus allowing us to control when flush/commit occurs

6. trait FlushingProcessor[K, V] extends AbstractProcessor[K, V] with OnInit with Flushing with ProcessorContextLogging

   

7. trait FlushingTransformer[K, V, K1, V1] extends Transformer[K, V, (K1, V1)] with Flushing