Guidelines for the Structured Logging Framework - Java Logging

Use the org.apache.spark.internal.SparkLoggerFactory to get the logger instance in Java code: Getting Logger Instance: Instead of using org.slf4j.LoggerFactory, use org.apache.spark.internal.SparkLoggerFactory to ensure structured logging.

import org.apache.spark.internal.SparkLogger; import org.apache.spark.internal.SparkLoggerFactory; private static final SparkLogger logger = SparkLoggerFactory.getLogger(JavaUtils.class);

Logging Messages with Variables: When logging messages with variables, wrap all the variables with MDCs and they will be automatically added to the Mapped Diagnostic Context (MDC).

import org.apache.spark.internal.LogKeys; import org.apache.spark.internal.MDC; logger.error("Unable to delete file for partition {}", MDC.of(LogKeys.PARTITION_ID$.MODULE$, i));

Constant String Messages: For logging constant string messages, use the standard logging methods.

logger.error("Failed to abort the writer after failing to write map output.", e);

If you want to output logs in java code through the structured log framework, you can define custom LogKey and use it in java code as follows:

// To add a custom LogKey, implement LogKey public static class CUSTOM_LOG_KEY implements LogKey { } import org.apache.spark.internal.MDC; logger.error("Unable to delete key {} for cache", MDC.of(CUSTOM_LOG_KEY, "key"));