org.elasticsearch.spark.sql.streaming

Class EsSinkMetadataLog

java.lang.Object

org.apache.spark.sql.execution.streaming.HDFSMetadataLog<T[]>

org.apache.spark.sql.execution.streaming.CompactibleFileStreamLog<EsSinkStatus>

org.elasticsearch.spark.sql.streaming.EsSinkMetadataLog

All Implemented Interfaces:

org.apache.spark.internal.Logging, org.apache.spark.sql.execution.streaming.MetadataLog<EsSinkStatus[]>

public class EsSinkMetadataLog
extends org.apache.spark.sql.execution.streaming.CompactibleFileStreamLog<EsSinkStatus>

Provides logic for managing batch ID committing as well as cleaning up state between batch executions for the EsSparkSqlStreamingSink

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.spark.sql.execution.streaming.HDFSMetadataLog

org.apache.spark.sql.execution.streaming.HDFSMetadataLog.FileContextManager, org.apache.spark.sql.execution.streaming.HDFSMetadataLog.FileManager, org.apache.spark.sql.execution.streaming.HDFSMetadataLog.FileSystemManager

Constructor Summary

Constructors

Constructor and Description
EsSinkMetadataLog(Settings settings, SparkSession sparkSession, String path)

Method Summary

Modifier and Type	Method and Description
scala.collection.Seq<EsSinkStatus>	compactLogs(scala.collection.Seq<EsSinkStatus> logs)
protected int	defaultCompactInterval()
protected long	fileCleanupDelayMs()
protected boolean	isDeletingExpiredLog()
static int	VERSION_NUMBER()

Methods inherited from class org.apache.spark.sql.execution.streaming.CompactibleFileStreamLog

add, allFiles, batchIdToPath, COMPACT_FILE_SUFFIX, compactInterval, deriveCompactInterval, deserialize, getAllValidBatches, getBatchIdFromFileName, getValidBatchesBeforeCompactionBatch, isBatchFile, isCompactionBatch, minBatchesToRetain, nextCompactionBatchId, org$apache$spark$sql$execution$streaming$CompactibleFileStreamLog$$formats, org$apache$spark$sql$execution$streaming$CompactibleFileStreamLog$$manifest, pathToBatchId, serialize

Methods inherited from class org.apache.spark.sql.execution.streaming.HDFSMetadataLog

batchFilesFilter, fileManager, get, get, get, getLatest, getOrderedBatchFiles, initializeLogIfNecessary, initializeLogIfNecessary, initializeLogIfNecessary$default$2, isTraceEnabled, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logName, logTrace, logTrace, logWarning, logWarning, metadataPath, org$apache$spark$internal$Logging$$log__$eq, org$apache$spark$internal$Logging$$log_, org$apache$spark$sql$execution$streaming$HDFSMetadataLog$$writeBatch, parseVersion, purge, purgeAfter, verifyBatchIds

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

EsSinkMetadataLog

public EsSinkMetadataLog(Settings settings,
                         SparkSession sparkSession,
                         String path)

Method Detail

VERSION_NUMBER

public static int VERSION_NUMBER()

fileCleanupDelayMs

protected long fileCleanupDelayMs()

Specified by:

fileCleanupDelayMs in class org.apache.spark.sql.execution.streaming.CompactibleFileStreamLog<EsSinkStatus>

isDeletingExpiredLog

protected boolean isDeletingExpiredLog()

Specified by:

isDeletingExpiredLog in class org.apache.spark.sql.execution.streaming.CompactibleFileStreamLog<EsSinkStatus>

defaultCompactInterval

protected int defaultCompactInterval()

Specified by:

defaultCompactInterval in class org.apache.spark.sql.execution.streaming.CompactibleFileStreamLog<EsSinkStatus>

compactLogs

public scala.collection.Seq<EsSinkStatus> compactLogs(scala.collection.Seq<EsSinkStatus> logs)

Specified by:

compactLogs in class org.apache.spark.sql.execution.streaming.CompactibleFileStreamLog<EsSinkStatus>