Packages

package root

Definition Classes: root

package org

Definition Classes: root

package apache

Definition Classes: org

package spark

Definition Classes: apache

package sql

Allows the execution of relational queries, including those expressed in SQL using Spark.

Definition Classes: spark

package execution

The physical execution component of Spark SQL.

The physical execution component of Spark SQL. Note that this is a private package. All classes in catalyst are considered an internal API to Spark SQL and are subject to change between minor releases.

Definition Classes: sql

package adaptive

Definition Classes: execution

package aggregate

Definition Classes: execution

package analysis

Definition Classes: execution

package bucketing

Definition Classes: execution

package columnar

Definition Classes: execution

package command

Definition Classes: execution

package datasources

Definition Classes: execution

package debug

Contains methods for debugging query execution.

Usage:

import org.apache.spark.sql.execution.debug._
sql("SELECT 1").debug()
sql("SELECT 1").debugCodegen()

or for streaming case (structured streaming):

import org.apache.spark.sql.execution.debug._
val query = df.writeStream.<...>.start()
query.debugCodegen()

Note that debug in structured streaming is not supported, because it doesn't make sense for streaming to execute batch once while main query is running concurrently.

Definition Classes: execution

package dynamicpruning

Definition Classes: execution

package exchange

Definition Classes: execution

package joins

Definition Classes: execution

package metric

Definition Classes: execution

package python

Definition Classes: execution

package r

Definition Classes: execution

package reuse

Definition Classes: execution

ReuseExchangeAndSubquery

package stat

Definition Classes: execution

package streaming

Definition Classes: execution

package ui

Definition Classes: execution

package vectorized

Definition Classes: execution

package window

Definition Classes: execution

org.apache.spark.sql.execution

reuse

package reuse

Ordering

Alphabetic

Visibility

Public
Protected

Value Members

case object ReuseExchangeAndSubquery extends Rule[SparkPlan] with Product with Serializable
Find out duplicated exchanges and subqueries in the whole spark plan including subqueries, then use the same exchange or subquery for all the references.
Find out duplicated exchanges and subqueries in the whole spark plan including subqueries, then use the same exchange or subquery for all the references.
Note that the Spark plan is a mutually recursive data structure: SparkPlan -> Expr -> Subquery -> SparkPlan -> Expr -> Subquery -> ... Therefore, in this rule, we recursively rewrite the exchanges and subqueries in a bottom-up way, in one go.

Packages

reuse

package reuse

Value Members

Ungrouped

reuse