org.elasticsearch.spark.sql.streaming

Class SparkSqlStreamingConfigs

java.lang.Object

org.elasticsearch.spark.sql.streaming.SparkSqlStreamingConfigs

public class SparkSqlStreamingConfigs
extends Object

Configurations specifically used for Spark Structured Streaming

Constructor Summary

Constructors

Constructor and Description
SparkSqlStreamingConfigs()

Method Summary

Modifier and Type	Method and Description
static String	constructCommitLogPath(Settings settings) Determines the location of the streaming commit log.
static String	ES_INTERNAL_APP_ID()
static String	ES_INTERNAL_APP_NAME()
static String	ES_INTERNAL_QUERY_NAME()
static String	ES_INTERNAL_SESSION_CHECKPOINT_LOCATION()
static String	ES_INTERNAL_USER_CHECKPOINT_LOCATION()
static long	ES_SINK_LOG_CLEANUP_DELAY_DEFAULT()
static String	ES_SINK_LOG_CLEANUP_DELAY()
static int	ES_SINK_LOG_COMPACT_INTERVAL_DEFAULT()
static String	ES_SINK_LOG_COMPACT_INTERVAL()
static boolean	ES_SINK_LOG_DELETION_DEFAULT()
static String	ES_SINK_LOG_DELETION()
static boolean	ES_SINK_LOG_ENABLE_DEFAULT()
static String	ES_SINK_LOG_ENABLE()
static String	ES_SINK_LOG_PATH()
static scala.Option<String>	getAppId(Settings settings) The ID of the current Spark application, if set
static scala.Option<String>	getAppName(Settings settings) The name of the current Spark application, if set
static int	getDefaultCompactInterval(Settings settings)
static long	getFileCleanupDelayMs(Settings settings) The number of milliseconds to wait before cleaning up compacted log files
static boolean	getIsDeletingExpiredLog(Settings settings)
static scala.Option<String>	getLogPath(Settings settings) The log path, if set, is the complete log path to be used for the commit log.
static scala.Option<String>	getQueryName(Settings settings) The name of the current Spark Streaming Query, if set
static scala.Option<String>	getSessionCheckpointLocation(Settings settings) The name of the default session checkpoint location, if set
static boolean	getSinkLogEnabled(Settings settings) Determines if we should use the commit log for writes, or if we should go without one.
static scala.Option<String>	getUserSpecifiedCheckpointLocation(Settings settings) The name of the user specified checkpoint location for the current Spark Streaming Query, if set

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

SparkSqlStreamingConfigs

public SparkSqlStreamingConfigs()

Method Detail

ES_SINK_LOG_ENABLE

public static String ES_SINK_LOG_ENABLE()

ES_SINK_LOG_ENABLE_DEFAULT

public static boolean ES_SINK_LOG_ENABLE_DEFAULT()

ES_SINK_LOG_PATH

public static String ES_SINK_LOG_PATH()

ES_INTERNAL_APP_NAME

public static String ES_INTERNAL_APP_NAME()

ES_INTERNAL_APP_ID

public static String ES_INTERNAL_APP_ID()

ES_INTERNAL_QUERY_NAME

public static String ES_INTERNAL_QUERY_NAME()

ES_INTERNAL_USER_CHECKPOINT_LOCATION

public static String ES_INTERNAL_USER_CHECKPOINT_LOCATION()

ES_INTERNAL_SESSION_CHECKPOINT_LOCATION

public static String ES_INTERNAL_SESSION_CHECKPOINT_LOCATION()

ES_SINK_LOG_CLEANUP_DELAY

public static String ES_SINK_LOG_CLEANUP_DELAY()

ES_SINK_LOG_CLEANUP_DELAY_DEFAULT

public static long ES_SINK_LOG_CLEANUP_DELAY_DEFAULT()

ES_SINK_LOG_DELETION

public static String ES_SINK_LOG_DELETION()

ES_SINK_LOG_DELETION_DEFAULT

public static boolean ES_SINK_LOG_DELETION_DEFAULT()

ES_SINK_LOG_COMPACT_INTERVAL

public static String ES_SINK_LOG_COMPACT_INTERVAL()

ES_SINK_LOG_COMPACT_INTERVAL_DEFAULT

public static int ES_SINK_LOG_COMPACT_INTERVAL_DEFAULT()

getSinkLogEnabled

public static boolean getSinkLogEnabled(Settings settings)

Determines if we should use the commit log for writes, or if we should go without one.

Parameters:

settings - connector settings

Returns:

true if we should use the commit log, false if we should not

constructCommitLogPath

public static String constructCommitLogPath(Settings settings)

Determines the location of the streaming commit log.

Parameters:

settings - connector settings

Returns:

the location to use as the commit log

getLogPath

public static scala.Option<String> getLogPath(Settings settings)

The log path, if set, is the complete log path to be used for the commit log.

Parameters:

settings - connector settings

Returns:

either Some log path or None

getAppName

public static scala.Option<String> getAppName(Settings settings)

The name of the current Spark application, if set

Parameters:

settings - connector settings

Returns:

Some name or None

getAppId

public static scala.Option<String> getAppId(Settings settings)

The ID of the current Spark application, if set

Parameters:

settings - connector settings

Returns:

Some id or None

getQueryName

public static scala.Option<String> getQueryName(Settings settings)

The name of the current Spark Streaming Query, if set

Parameters:

settings - connector settings

Returns:

Some query name or None

getUserSpecifiedCheckpointLocation

public static scala.Option<String> getUserSpecifiedCheckpointLocation(Settings settings)

The name of the user specified checkpoint location for the current Spark Streaming Query, if set

Parameters:

settings - connector settings

Returns:

Some checkpoint location or None

getSessionCheckpointLocation

public static scala.Option<String> getSessionCheckpointLocation(Settings settings)

The name of the default session checkpoint location, if set

Parameters:

settings - connector settings

Returns:

Some checkpoint location or None

getFileCleanupDelayMs

public static long getFileCleanupDelayMs(Settings settings)

The number of milliseconds to wait before cleaning up compacted log files

Parameters:

settings - connector settings

Returns:

time in millis if set, or the default delay

getIsDeletingExpiredLog

public static boolean getIsDeletingExpiredLog(Settings settings)

Parameters:

settings - connector settings

Returns:

getDefaultCompactInterval

public static int getDefaultCompactInterval(Settings settings)

Parameters:

settings - connector settings

Returns: