Type Members

1.  trait LogKey extends AnyRef

   All structured logging keys used in MDC must be extends LogKey

   All structured logging keys used in MDC must be extends LogKey

   LogKeys serve as identifiers for mapped diagnostic contexts (MDC) within logs. Follow these guidelines when adding a new LogKey:

   - Define all structured logging keys in LogKeys.java, and sort them alphabetically for ease of search.

   - Use UPPER_SNAKE_CASE for key names.

   - Key names should be both simple and broad, yet include specific identifiers like STAGE_ID, TASK_ID, and JOB_ID when needed for clarity. For instance, use MAX_ATTEMPTS as a general key instead of creating separate keys for each scenario such as EXECUTOR_STATE_SYNC_MAX_ATTEMPTS and MAX_TASK_FAILURES. This balances simplicity with the detail needed for effective logging.

   - Use abbreviations in names if they are widely understood, such as APP_ID for APPLICATION_ID, and K8S for KUBERNETES.

   - For time-related keys, use milliseconds as the unit of time.

2.  sealed final class LogKeys extends Enum[LogKeys] with LogKey

   Various keys used for mapped diagnostic contexts(MDC) in logging.

   Various keys used for mapped diagnostic contexts(MDC) in logging. All structured logging keys should be defined here for standardization.

3.  final class MDC extends Record

   Mapped Diagnostic Context (MDC) that will be used in log messages.

   Mapped Diagnostic Context (MDC) that will be used in log messages. The values of the MDC will be inline in the log message, while the key-value pairs will be part of the ThreadContext.

4.  class SparkLogger extends AnyRef

   Guidelines for the Structured Logging Framework - Java Logging

   Guidelines for the Structured Logging Framework - Java Logging

   Use the org.apache.spark.internal.SparkLoggerFactory to get the logger instance in Java code: Getting Logger Instance: Instead of using org.slf4j.LoggerFactory, use org.apache.spark.internal.SparkLoggerFactory to ensure structured logging.

   import org.apache.spark.internal.SparkLogger; import org.apache.spark.internal.SparkLoggerFactory; private static final SparkLogger logger = SparkLoggerFactory.getLogger(JavaUtils.class);

   Logging Messages with Variables: When logging messages with variables, wrap all the variables with MDCs and they will be automatically added to the Mapped Diagnostic Context (MDC).

   import org.apache.spark.internal.LogKeys; import org.apache.spark.internal.MDC; logger.error("Unable to delete file for partition {}", MDC.of(LogKeys.PARTITION_ID, i));

   Constant String Messages: For logging constant string messages, use the standard logging methods.

   logger.error("Failed to abort the writer after failing to write map output.", e);

   If you want to output logs in java code through the structured log framework, you can define custom LogKey and use it in java code as follows:

   // Add a CustomLogKeys, implement LogKey public enum CustomLogKeys implements LogKey { CUSTOM_LOG_KEY } import org.apache.spark.internal.MDC; logger.error("Unable to delete key {} for cache", MDC.of(CUSTOM_LOG_KEY, "key"));

5.  class SparkLoggerFactory extends AnyRef