org.elasticsearch.spark.sql.streaming

Class EsCommitProtocol

java.lang.Object

org.elasticsearch.spark.sql.streaming.EsCommitProtocol

All Implemented Interfaces:

Serializable

public class EsCommitProtocol
extends Object
implements scala.Serializable

Defines job-side and task-side logic for committing batches of writes to Elasticsearch for Spark Structured Streaming

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
EsCommitProtocol(org.apache.spark.sql.execution.streaming.MetadataLog<EsSinkStatus[]> commitLog)

Method Summary

Modifier and Type	Method and Description
void	abortJob(JobState jobState) Abort this job execution.
void	abortTask(TaskState taskState) Abort this task execution
void	commitJob(JobState jobState, scala.collection.Seq<TaskCommit> taskCommits) Commit the results of each of the tasks to the execution log.
org.apache.spark.sql.execution.streaming.MetadataLog<EsSinkStatus[]>	commitLog()
TaskCommit	commitTask(TaskState taskState) Commit this task's transaction.
long	executionStart()
void	initJob(JobState jobState) Prepare the driver side of the commit protocol.
void	initTask(TaskState taskState) Initialize the executor side of the commit protocol.
Log	logger()
void	recordSeen() Signal that a record has been processed in this transaction.
long	recordsWritten()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

EsCommitProtocol

public EsCommitProtocol(org.apache.spark.sql.execution.streaming.MetadataLog<EsSinkStatus[]> commitLog)

Method Detail

abortJob

public void abortJob(JobState jobState)

Abort this job execution.

Parameters:

jobState - the current Job information

abortTask

public void abortTask(TaskState taskState)

Abort this task execution

Parameters:

taskState - the current Task information

commitJob

public void commitJob(JobState jobState,
                      scala.collection.Seq<TaskCommit> taskCommits)

Commit the results of each of the tasks to the execution log.

Parameters:

jobState - the current Job information

taskCommits - the results from each of the tasks

commitLog

public org.apache.spark.sql.execution.streaming.MetadataLog<EsSinkStatus[]> commitLog()

commitTask

public TaskCommit commitTask(TaskState taskState)

Commit this task's transaction.

Parameters:

taskState - the current Task information

Returns:

a summary of this Task's actions as a commit message to be added to the log

executionStart

public long executionStart()

initJob

public void initJob(JobState jobState)

Prepare the driver side of the commit protocol.

Parameters:

jobState - the current Job information

initTask

public void initTask(TaskState taskState)

Initialize the executor side of the commit protocol.

Parameters:

taskState - the current Task information

logger

public Log logger()

recordSeen

public void recordSeen()

Signal that a record has been processed in this transaction.

recordsWritten

public long recordsWritten()