Allows the execution of relational queries, including those expressed in SQL using Spark.
Allows the execution of relational queries, including those expressed in SQL using Spark.
The physical execution component of Spark SQL.
The physical execution component of Spark SQL. Note that this is a private package. All classes in catalyst are considered an internal API to Spark SQL and are subject to change between minor releases.
Contains methods for debugging query execution.
Contains methods for debugging query execution.
Usage:
import org.apache.spark.sql.execution.debug._ sql("SELECT 1").debug() sql("SELECT 1").debugCodegen()
or for streaming case (structured streaming):
import org.apache.spark.sql.execution.debug._ val query = df.writeStream.<...>.start() query.debugCodegen()
Note that debug in structured streaming is not supported, because it doesn't make sense for streaming to execute batch once while main query is running concurrently.