Function that produces a new instance of the SessionStateBuilder.
Function that produces a new instance of the SessionStateBuilder. This is used by the SessionState's clone functionality. Make sure to override this when implementing your own SessionStateBuilder.
Logical query plan analyzer for resolving unresolved attributes and relations.
Logical query plan analyzer for resolving unresolved attributes and relations.
Note: this depends on the conf
and catalog
fields.
Build the SessionState.
Catalog for managing table and database states.
Catalog for managing table and database states. If there is a pre-existing catalog, the state of that catalog (temp tables & current database) will be copied into the new catalog.
Note: this depends on the conf
, functionRegistry
and sqlParser
fields.
SQL-specific key-value configurations.
SQL-specific key-value configurations.
These either get cloned from a pre-existing instance or newly created. The conf is always merged with its SparkConf.
Function used to make clones of the session state.
Function used to make clones of the session state.
Create a query execution object.
Create a query execution object.
Custom check rules to add to the Analyzer.
Custom check rules to add to the Analyzer. Prefer overriding this instead of creating your own Analyzer.
Note that this may NOT depend on the analyzer
function.
Custom operator optimization rules to add to the Optimizer.
Custom operator optimization rules to add to the Optimizer. Prefer overriding this instead of creating your own Optimizer.
Note that this may NOT depend on the optimizer
function.
Custom strategies to add to the planner.
Custom strategies to add to the planner. Prefer overriding this instead of creating your own Planner.
Note that this may NOT depend on the planner
function.
Custom post resolution rules to add to the Analyzer.
Custom post resolution rules to add to the Analyzer. Prefer overriding this instead of creating your own Analyzer.
Note that this may NOT depend on the analyzer
function.
Custom resolution rules to add to the Analyzer.
Custom resolution rules to add to the Analyzer. Prefer overriding this instead of creating your own Analyzer.
Note that this may NOT depend on the analyzer
function.
Experimental methods that can be used to define custom optimization rules and custom planning strategies.
Experimental methods that can be used to define custom optimization rules and custom planning strategies.
This either gets cloned from a pre-existing version or newly created.
Session extensions defined in the SparkSession.
Session extensions defined in the SparkSession.
Internal catalog managing functions registered by the user.
Internal catalog managing functions registered by the user.
This either gets cloned from a pre-existing version or cloned from the built-in registry.
An interface to register custom org.apache.spark.sql.util.QueryExecutionListeners that listen for execution metrics.
An interface to register custom org.apache.spark.sql.util.QueryExecutionListeners that listen for execution metrics.
This gets cloned from parent if available, otherwise is a new instance is created.
Extract entries from SparkConf
and put them in the SQLConf
Extract entries from SparkConf
and put them in the SQLConf
Logical query plan optimizer.
Logical query plan optimizer.
Note: this depends on the conf
, catalog
and experimentalMethods
fields.
Planner that converts optimized logical plans to physical plans.
Planner that converts optimized logical plans to physical plans.
Note: this depends on the conf
and experimentalMethods
fields.
ResourceLoader that is used to load function resources and jars.
ResourceLoader that is used to load function resources and jars.
Parser that extracts expressions, plans, table identifiers etc.
Parser that extracts expressions, plans, table identifiers etc. from SQL texts.
Note: this depends on the conf
field.
Interface to start and stop streaming queries.
Interface to start and stop streaming queries.
Interface exposed to the user for registering user-defined functions.
Interface exposed to the user for registering user-defined functions.
Note 1: The user-defined functions must be deterministic.
Note 2: This depends on the functionRegistry
field.
Builder class that coordinates construction of a new SessionState.
The builder explicitly defines all components needed by the session state, and creates a session state when
build
is called. Components should only be initialized once. This is not a problem for most components as they are only used in thebuild
function. However some components (conf
,catalog
,functionRegistry
,experimentalMethods
&sqlParser
) are as dependencies for other components and are shared as a result. These components are defined as lazy vals to make sure the component is created only once.A developer can modify the builder by providing custom versions of components, or by using the hooks provided for the analyzer, optimizer & planner. There are some dependencies between the components (they are documented per dependency), a developer should respect these when making modifications in order to prevent initialization problems.
A parent SessionState can be used to initialize the new SessionState. The new session state will clone the parent sessions state's
conf
,functionRegistry
,experimentalMethods
andcatalog
fields. Note that the state is cloned whenbuild
is called, and not before.