org.apache.kafka.connect.sink

Class SinkTask

java.lang.Object

org.apache.kafka.connect.sink.SinkTask

All Implemented Interfaces:

Task

@InterfaceStability.Unstable
public abstract class SinkTask
extends Object
implements Task

SinkTask is a Task takes records loaded from Kafka and sends them to another system. In addition to the basic put(java.util.Collection<org.apache.kafka.connect.sink.SinkRecord>) interface, SinkTasks must also implement flush(java.util.Map<org.apache.kafka.common.TopicPartition, org.apache.kafka.clients.consumer.OffsetAndMetadata>) to support offset commits.

Field Summary

Fields

Modifier and Type	Field and Description
protected SinkTaskContext	context
static String	TOPICS_CONFIG The configuration key that provides the list of topics that are inputs for this SinkTask.

Constructor Summary

Constructors

Constructor and Description
SinkTask()

Method Summary

Methods

Modifier and Type	Method and Description
abstract void	flush(Map<org.apache.kafka.common.TopicPartition,org.apache.kafka.clients.consumer.OffsetAndMetadata> offsets) Flush all records that have been put(java.util.Collection<org.apache.kafka.connect.sink.SinkRecord>) for the specified topic-partitions.
void	initialize(SinkTaskContext context)
void	onPartitionsAssigned(Collection<org.apache.kafka.common.TopicPartition> partitions) The SinkTask use this method to create writers for newly assigned partitions in case of partition re-assignment.
void	onPartitionsRevoked(Collection<org.apache.kafka.common.TopicPartition> partitions) The SinkTask use this method to close writers and commit offsets for partitions that are longer assigned to the SinkTask.
abstract void	put(Collection<SinkRecord> records) Put the records in the sink.
abstract void	start(Map<String,String> props) Start the Task.
abstract void	stop() Perform any cleanup to stop this task.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.kafka.connect.connector.Task

version

Field Detail

TOPICS_CONFIG

public static final String TOPICS_CONFIG

The configuration key that provides the list of topics that are inputs for this SinkTask.

See Also:

Constant Field Values

context

protected SinkTaskContext context

Constructor Detail

SinkTask

public SinkTask()

Method Detail

initialize

public void initialize(SinkTaskContext context)

start

public abstract void start(Map<String,String> props)

Start the Task. This should handle any configuration parsing and one-time setup of the task.

Specified by:

start in interface Task

Parameters:

props - initial configuration

put

public abstract void put(Collection<SinkRecord> records)

Put the records in the sink. Usually this should send the records to the sink asynchronously and immediately return. If this operation fails, the SinkTask may throw a RetriableException to indicate that the framework should attempt to retry the same call again. Other exceptions will cause the task to be stopped immediately. SinkTaskContext.timeout(long) can be used to set the maximum time before the batch will be retried.

Parameters:

records - the set of records to send

flush

public abstract void flush(Map<org.apache.kafka.common.TopicPartition,org.apache.kafka.clients.consumer.OffsetAndMetadata> offsets)

Flush all records that have been put(java.util.Collection<org.apache.kafka.connect.sink.SinkRecord>) for the specified topic-partitions. The offsets are provided for convenience, but could also be determined by tracking all offsets included in the SinkRecords passed to put(java.util.Collection<org.apache.kafka.connect.sink.SinkRecord>).

Parameters:

offsets - mapping of TopicPartition to committed offset

onPartitionsAssigned

public void onPartitionsAssigned(Collection<org.apache.kafka.common.TopicPartition> partitions)

The SinkTask use this method to create writers for newly assigned partitions in case of partition re-assignment. In partition re-assignment, some new partitions may be assigned to the SinkTask. The SinkTask needs to create writers and perform necessary recovery for the newly assigned partitions. This method will be called after partition re-assignment completes and before the SinkTask starts fetching data.

Parameters:

partitions - The list of partitions that are now assigned to the task (may include partitions previously assigned to the task)

onPartitionsRevoked

public void onPartitionsRevoked(Collection<org.apache.kafka.common.TopicPartition> partitions)

The SinkTask use this method to close writers and commit offsets for partitions that are longer assigned to the SinkTask. This method will be called before a rebalance operation starts and after the SinkTask stops fetching data.

Parameters:

partitions - The list of partitions that were assigned to the consumer on the last rebalance

stop

public abstract void stop()

Perform any cleanup to stop this task. In SinkTasks, this method is invoked only once outstanding calls to other methods have completed (e.g., put(Collection) has returned) and a final flush(Map) and offset commit has completed. Implementations of this method should only need to perform final cleanup operations, such as closing network connections to the sink system.

Specified by:

stop in interface Task