Packages

package root

Definition Classes: root

package org

Definition Classes: root

package apache

Definition Classes: org

package spark

Definition Classes: apache

package sql

Allows the execution of relational queries, including those expressed in SQL using Spark.

Definition Classes: spark

package execution

The physical execution component of Spark SQL.

The physical execution component of Spark SQL. Note that this is a private package. All classes in catalyst are considered an internal API to Spark SQL and are subject to change between minor releases.

Definition Classes: sql

package adaptive

Definition Classes: execution

package aggregate

Definition Classes: execution

package analysis

Definition Classes: execution

package arrow

Definition Classes: execution

package bucketing

Definition Classes: execution

package columnar

Definition Classes: execution

package command

Definition Classes: execution

package datasources

Definition Classes: execution

package debug

Contains methods for debugging query execution.

Usage:

import org.apache.spark.sql.execution.debug._
sql("SELECT 1").debug()
sql("SELECT 1").debugCodegen()

or for streaming case (structured streaming):

import org.apache.spark.sql.execution.debug._
val query = df.writeStream.<...>.start()
query.debugCodegen()

Note that debug in structured streaming is not supported, because it doesn't make sense for streaming to execute batch once while main query is running concurrently.

Definition Classes: execution

package dynamicpruning

Definition Classes: execution

package exchange

Definition Classes: execution

package joins

Definition Classes: execution

package metric

Definition Classes: execution

package python

Definition Classes: execution

package r

Definition Classes: execution

ArrowRRunner

MapPartitionsRWrapper

package reuse

Definition Classes: execution

package stat

Definition Classes: execution

package streaming

Definition Classes: execution

package ui

Definition Classes: execution

package vectorized

Definition Classes: execution

package window

Definition Classes: execution

org.apache.spark.sql.execution

r

package r

Type Members

class ArrowRRunner extends BaseRRunner[Iterator[InternalRow], ColumnarBatch]
Similar to ArrowPythonRunner, but exchange data with R worker via Arrow stream.
case class MapPartitionsRWrapper(func: Array[Byte], packageNames: Array[Byte], broadcastVars: Array[Broadcast[AnyRef]], inputSchema: StructType, outputSchema: StructType) extends (Iterator[Any]) => Iterator[Any] with Product with Serializable
A function wrapper that applies the given R function to each partition.

Packages

r

package r

Type Members

Ungrouped

r