Type Members

1.  type DataFrame = Dataset[Row]
2.  class ExperimentalMethods extends AnyRef

   :: Experimental :: Holder for experimental methods for the bravest.

   :: Experimental :: Holder for experimental methods for the bravest. We make NO guarantee about the stability regarding binary compatibility and source compatibility of methods here.

   spark.experimental.extraStrategies += ...

   Annotations

   @Experimental() @Unstable()

   Since

   1.3.0

3.  trait ExtendedExplainGenerator extends AnyRef

   A trait for a session extension to implement that provides addition explain plan information.

   A trait for a session extension to implement that provides addition explain plan information.

   Annotations

   @DeveloperApi() @Since("4.0.0")

4.  class SparkSessionExtensions extends AnyRef

   :: Experimental :: Holder for injection points to the SparkSession.

   :: Experimental :: Holder for injection points to the SparkSession. We make NO guarantee about the stability regarding binary compatibility and source compatibility of methods here.

   This current provides the following extension points:

   • Analyzer Rules.
   • Check Analysis Rules.
   • Cache Plan Normalization Rules.
   • Optimizer Rules.
   • Pre CBO Rules.
   • Planning Strategies.
   • Customized Parser.
   • (External) Catalog listeners.
   • Columnar Rules.
   • Adaptive Query Post Planner Strategy Rules.
   • Adaptive Query Stage Preparation Rules.
   • Adaptive Query Execution Runtime Optimizer Rules.
   • Adaptive Query Stage Optimizer Rules.

   The extensions can be used by calling withExtensions on the SparkSession.Builder, for example:

   SparkSession.builder()
     .master("...")
     .config("...", true)
     .withExtensions { extensions =>
       extensions.injectResolutionRule { session =>
         ...
       }
       extensions.injectParser { (session, parser) =>
         ...
       }
     }
     .getOrCreate()

   The extensions can also be used by setting the Spark SQL configuration property spark.sql.extensions. Multiple extensions can be set using a comma-separated list. For example:

   SparkSession.builder()
     .master("...")
     .config("spark.sql.extensions", "org.example.MyExtensions,org.example.YourExtensions")
     .getOrCreate()
   
   class MyExtensions extends Function1[SparkSessionExtensions, Unit] {
     override def apply(extensions: SparkSessionExtensions): Unit = {
       extensions.injectResolutionRule { session =>
         ...
       }
       extensions.injectParser { (session, parser) =>
         ...
       }
     }
   }
   
   class YourExtensions extends SparkSessionExtensionsProvider {
     override def apply(extensions: SparkSessionExtensions): Unit = {
       extensions.injectResolutionRule { session =>
         ...
       }
       extensions.injectFunction(...)
     }
   }

   Note that none of the injected builders should assume that the SparkSession is fully initialized and should not touch the session's internals (e.g. the SessionState).

   Annotations

   @DeveloperApi() @Experimental() @Unstable()

5.  trait SparkSessionExtensionsProvider extends (SparkSessionExtensions) => Unit

   Base trait for implementations used by SparkSessionExtensions

   Base trait for implementations used by SparkSessionExtensions

   For example, now we have an external function named Age to register as an extension for SparkSession:

   package org.apache.spark.examples.extensions
   
   import org.apache.spark.sql.catalyst.expressions.{CurrentDate, Expression, RuntimeReplaceable, SubtractDates}
   
   case class Age(birthday: Expression, child: Expression) extends RuntimeReplaceable {
   
     def this(birthday: Expression) = this(birthday, SubtractDates(CurrentDate(), birthday))
     override def exprsReplaced: Seq[Expression] = Seq(birthday)
     override protected def withNewChildInternal(newChild: Expression): Expression = copy(newChild)
   }

   We need to create our extension which inherits SparkSessionExtensionsProvider Example:

   package org.apache.spark.examples.extensions
   
   import org.apache.spark.sql.{SparkSessionExtensions, SparkSessionExtensionsProvider}
   import org.apache.spark.sql.catalyst.FunctionIdentifier
   import org.apache.spark.sql.catalyst.expressions.{Expression, ExpressionInfo}
   
   class MyExtensions extends SparkSessionExtensionsProvider {
     override def apply(v1: SparkSessionExtensions): Unit = {
       v1.injectFunction(
         (new FunctionIdentifier("age"),
           new ExpressionInfo(classOf[Age].getName, "age"),
           (children: Seq[Expression]) => new Age(children.head)))
     }
   }

   Then, we can inject MyExtensions in three ways,

   • withExtensions of SparkSession.Builder
   • Config - spark.sql.extensions
   • java.util.ServiceLoader - Add to src/main/resources/META-INF/services/org.apache.spark.sql.SparkSessionExtensionsProvider

   Annotations

   @DeveloperApi() @Since("3.2.0")

   Since

   3.2.0

   See also

   SparkSessionExtensions

   SparkSession.Builder

   java.util.ServiceLoader