Package

org.apache.spark.sql.catalyst

expressions

Permalink

package expressions

A set of classes that can be used to represent trees of relational expressions. A key goal of the expression library is to hide the details of naming and scoping from developers who want to manipulate trees of relational operators. As such, the library defines a special type of expression, a NamedExpression in addition to the standard collection of expressions.

Standard Expressions

A library of standard expressions (e.g., Add, EqualTo), aggregates (e.g., SUM, COUNT), and other computations (e.g. UDFs). Each expression type is capable of determining its output schema as a function of its children's output schema.

Named Expressions

Some expression are named and thus can be referenced by later operators in the dataflow graph. The two types of named expressions are AttributeReferences and Aliases. AttributeReferences refer to attributes of the input tuple for a given operator and form the leaves of some expression trees. Aliases assign a name to intermediate computations. For example, in the SQL statement SELECT a+b AS c FROM ..., the expressions a and b would be represented by AttributeReferences and c would be represented by an Alias.

During analysis, all named expressions are assigned a globally unique expression id, which can be used for equality comparisons. While the original names are kept around for debugging purposes, they should never be used to check if two attributes refer to the same value, as plan transformations can result in the introduction of naming ambiguity. For example, consider a plan that contains subqueries, both of which are reading from the same table. If an optimization removes the subqueries, scoping information would be destroyed, eliminating the ability to reason about which subquery produced a given attribute.

Evaluation

The result of expressions can be evaluated using the Expression.apply(Row) method.

Linear Supertypes
AnyRef, Any
Ordering
  1. Alphabetic
  2. By inheritance
Inherited
  1. expressions
  2. AnyRef
  3. Any
  1. Hide All
  2. Show all
Visibility
  1. Public
  2. All

Type Members

  1. case class Abs(child: Expression) extends UnaryExpression with ExpectsInputTypes with Product with Serializable

    Permalink

    A function that get the absolute value of the numeric value.

    A function that get the absolute value of the numeric value.

    Annotations
    @ExpressionDescription()
  2. case class Acos(child: Expression) extends UnaryMathExpression with Product with Serializable

    Permalink
  3. case class Add(left: Expression, right: Expression) extends BinaryArithmetic with Product with Serializable

    Permalink
  4. case class AddMonths(startDate: Expression, numMonths: Expression) extends BinaryExpression with ImplicitCastInputTypes with Product with Serializable

    Permalink

    Returns the date that is num_months after start_date.

  5. case class Alias(child: Expression, name: String)(exprId: ExprId = NamedExpression.newExprId, qualifiers: Seq[String] = Nil, explicitMetadata: Option[Metadata] = None) extends UnaryExpression with NamedExpression with Product with Serializable

    Permalink

    Used to assign a new name to a computation.

    Used to assign a new name to a computation. For example the SQL expression "1 + 1 AS a" could be represented as follows: Alias(Add(Literal(1), Literal(1)), "a")()

    Note that exprId and qualifiers are in a separate parameter list because we only pattern match on child and name.

    child

    the computation being performed

    name

    the name to be associated with the result of computing child.

    exprId

    A globally unique id used to check if an AttributeReference refers to this alias. Auto-assigned if left blank.

    explicitMetadata

    Explicit metadata associated with this alias that overwrites child's.

  6. case class And(left: Expression, right: Expression) extends BinaryOperator with Predicate with Product with Serializable

    Permalink
  7. case class ArrayContains(left: Expression, right: Expression) extends BinaryExpression with ImplicitCastInputTypes with Product with Serializable

    Permalink

    Checks if the array (left) has the element (right)

  8. case class Ascii(child: Expression) extends UnaryExpression with ImplicitCastInputTypes with Product with Serializable

    Permalink

    Returns the numeric value of the first character of str.

  9. case class Asin(child: Expression) extends UnaryMathExpression with Product with Serializable

    Permalink
  10. case class AssertNotNull(child: Expression, walkedTypePath: Seq[String]) extends UnaryExpression with Product with Serializable

    Permalink

    Asserts that input values of a non-nullable child expression are not null.

    Asserts that input values of a non-nullable child expression are not null.

    Note that there are cases where child.nullable == true, while we still needs to add this assertion. Consider a nullable column s whose data type is a struct containing a non-nullable Int field named i. Expression s.i is nullable because s can be null. However, for all non-null s, s.i can't be null.

  11. case class AtLeastNNonNulls(n: Int, children: Seq[Expression]) extends Expression with Predicate with Product with Serializable

    Permalink

    A predicate that is evaluated to be true if there are at least n non-null and non-NaN values.

  12. case class Atan(child: Expression) extends UnaryMathExpression with Product with Serializable

    Permalink
  13. case class Atan2(left: Expression, right: Expression) extends BinaryMathExpression with Product with Serializable

    Permalink
  14. abstract class Attribute extends LeafExpression with NamedExpression

    Permalink
  15. class AttributeEquals extends AnyRef

    Permalink
    Attributes
    protected
  16. class AttributeMap[A] extends Map[Attribute, A] with Serializable

    Permalink
  17. case class AttributeReference(name: String, dataType: DataType, nullable: Boolean = true, metadata: Metadata = Metadata.empty)(exprId: ExprId = NamedExpression.newExprId, qualifiers: Seq[String] = Nil) extends Attribute with Unevaluable with Product with Serializable

    Permalink

    A reference to an attribute produced by another operator in the tree.

    A reference to an attribute produced by another operator in the tree.

    name

    The name of this attribute, should only be used during analysis or for debugging.

    dataType

    The DataType of this attribute.

    nullable

    True if null is a valid value for this attribute.

    metadata

    The metadata of this attribute.

    exprId

    A globally unique id used to check if different AttributeReferences refer to the same attribute.

    qualifiers

    a list of strings that can be used to referred to this attribute in a fully qualified way. Consider the examples tableName.name, subQueryAlias.name. tableName and subQueryAlias are possible qualifiers.

  18. implicit class AttributeSeq extends AnyRef

    Permalink

    Helper functions for working with Seq[Attribute].

  19. class AttributeSet extends Traversable[Attribute] with Serializable

    Permalink

    A Set designed to hold AttributeReference objects, that performs equality checking using expression id instead of standard java equality.

    A Set designed to hold AttributeReference objects, that performs equality checking using expression id instead of standard java equality. Using expression id means that these sets will correctly test for membership, even when the AttributeReferences in question differ cosmetically (e.g., the names have different capitalizations).

    Note that we do not override equality for Attribute references as it is really weird when AttributeReference("a"...) == AttrributeReference("b", ...). This tactic leads to broken tests, and also makes doing transformations hard (we always try keep older trees instead of new ones when the transformation was a no-op).

  20. case class Base64(child: Expression) extends UnaryExpression with ImplicitCastInputTypes with Product with Serializable

    Permalink

    Converts the argument from binary to a base 64 string.

  21. trait BaseGenericInternalRow extends InternalRow

    Permalink

    An extended version of InternalRow that implements all special getters, toString and equals/hashCode by genericGet.

  22. case class Bin(child: Expression) extends UnaryExpression with Serializable with ImplicitCastInputTypes with Product

    Permalink
  23. abstract class BinaryArithmetic extends BinaryOperator

    Permalink
  24. abstract class BinaryComparison extends BinaryOperator with Predicate

    Permalink
  25. abstract class BinaryExpression extends Expression

    Permalink

    An expression with two inputs and one output.

    An expression with two inputs and one output. The output is by default evaluated to null if any input is evaluated to null.

  26. abstract class BinaryMathExpression extends BinaryExpression with Serializable with ImplicitCastInputTypes

    Permalink

    A binary expression specifically for math functions that take two Doubles as input and returns a Double.

  27. abstract class BinaryOperator extends BinaryExpression with ExpectsInputTypes

    Permalink

    A BinaryExpression that is an operator, with two properties:

    A BinaryExpression that is an operator, with two properties:

    1. The string representation is "x symbol y", rather than "funcName(x, y)". 2. Two inputs are expected to the be same type. If the two inputs have different types, the analyzer will find the tightest common type and do the proper type casting.

  28. case class BitwiseAnd(left: Expression, right: Expression) extends BinaryArithmetic with Product with Serializable

    Permalink

    A function that calculates bitwise and(&) of two numbers.

    A function that calculates bitwise and(&) of two numbers.

    Code generation inherited from BinaryArithmetic.

  29. case class BitwiseNot(child: Expression) extends UnaryExpression with ExpectsInputTypes with Product with Serializable

    Permalink

    A function that calculates bitwise not(~) of a number.

  30. case class BitwiseOr(left: Expression, right: Expression) extends BinaryArithmetic with Product with Serializable

    Permalink

    A function that calculates bitwise or(|) of two numbers.

    A function that calculates bitwise or(|) of two numbers.

    Code generation inherited from BinaryArithmetic.

  31. case class BitwiseXor(left: Expression, right: Expression) extends BinaryArithmetic with Product with Serializable

    Permalink

    A function that calculates bitwise xor of two numbers.

    A function that calculates bitwise xor of two numbers.

    Code generation inherited from BinaryArithmetic.

  32. case class BoundReference(ordinal: Int, dataType: DataType, nullable: Boolean) extends LeafExpression with NamedExpression with Product with Serializable

    Permalink

    A bound reference points to a specific slot in the input tuple, allowing the actual value to be retrieved more efficiently.

    A bound reference points to a specific slot in the input tuple, allowing the actual value to be retrieved more efficiently. However, since operations like column pruning can change the layout of intermediate tuples, BindReferences should be run after all such transformations.

  33. case class CaseKeyWhen(key: Expression, branches: Seq[Expression]) extends Expression with CaseWhenLike with CodegenFallback with Product with Serializable

    Permalink

    Case statements of the form "CASE a WHEN b THEN c [WHEN d THEN e]* [ELSE f] END".

    Case statements of the form "CASE a WHEN b THEN c [WHEN d THEN e]* [ELSE f] END". Refer to this link for the corresponding semantics: https://cwiki.apache.org/confluence/display/Hive/LanguageManual+UDF#LanguageManualUDF-ConditionalFunctions

  34. case class CaseWhen(branches: Seq[Expression]) extends Expression with CaseWhenLike with CodegenFallback with Product with Serializable

    Permalink

    Case statements of the form "CASE WHEN a THEN b [WHEN c THEN d]* [ELSE e] END".

    Case statements of the form "CASE WHEN a THEN b [WHEN c THEN d]* [ELSE e] END". Refer to this link for the corresponding semantics: https://cwiki.apache.org/confluence/display/Hive/LanguageManual+UDF#LanguageManualUDF-ConditionalFunctions

  35. trait CaseWhenLike extends Expression

    Permalink
  36. case class Cast(child: Expression, dataType: DataType) extends UnaryExpression with CodegenFallback with Product with Serializable

    Permalink

    Cast the child expression to the target data type.

  37. case class Cbrt(child: Expression) extends UnaryMathExpression with Product with Serializable

    Permalink
  38. case class Ceil(child: Expression) extends UnaryMathExpression with Product with Serializable

    Permalink
  39. case class CheckOverflow(child: Expression, dataType: DecimalType) extends UnaryExpression with Product with Serializable

    Permalink

    Rounds the decimal to given scale and check whether the decimal can fit in provided precision or not, returns null if not.

  40. case class Coalesce(children: Seq[Expression]) extends Expression with Product with Serializable

    Permalink

    An expression that is evaluated to the first non-null input.

    An expression that is evaluated to the first non-null input.

    coalesce(1, 2) => 1
    coalesce(null, 1, 2) => 1
    coalesce(null, null, 2) => 2
    coalesce(null, null, null) => null
  41. case class Concat(children: Seq[Expression]) extends Expression with ImplicitCastInputTypes with Product with Serializable

    Permalink

    An expression that concatenates multiple input strings into a single string.

    An expression that concatenates multiple input strings into a single string. If any input is null, concat returns null.

  42. case class ConcatWs(children: Seq[Expression]) extends Expression with ImplicitCastInputTypes with Product with Serializable

    Permalink

    An expression that concatenates multiple input strings or array of strings into a single string, using a given separator (the first child).

    An expression that concatenates multiple input strings or array of strings into a single string, using a given separator (the first child).

    Returns null if the separator is null. Otherwise, concat_ws skips all null values.

  43. case class Contains(left: Expression, right: Expression) extends BinaryExpression with StringPredicate with Product with Serializable

    Permalink

    A function that returns true if the string left contains the string right.

  44. case class Conv(numExpr: Expression, fromBaseExpr: Expression, toBaseExpr: Expression) extends TernaryExpression with ImplicitCastInputTypes with Product with Serializable

    Permalink

    Convert a num from one base to another

    Convert a num from one base to another

    numExpr

    the number to be converted

    fromBaseExpr

    from which base

    toBaseExpr

    to which base

  45. case class Cos(child: Expression) extends UnaryMathExpression with Product with Serializable

    Permalink
  46. case class Cosh(child: Expression) extends UnaryMathExpression with Product with Serializable

    Permalink
  47. case class Crc32(child: Expression) extends UnaryExpression with ImplicitCastInputTypes with Product with Serializable

    Permalink

    A function that computes a cyclic redundancy check value and returns it as a bigint For input of type BinaryType

  48. case class CreateArray(children: Seq[Expression]) extends Expression with Product with Serializable

    Permalink

    Returns an Array containing the evaluation of all children expressions.

  49. case class CreateExternalRow(children: Seq[Expression]) extends Expression with Product with Serializable

    Permalink

    Constructs a new external row, using the result of evaluating the specified expressions as content.

    Constructs a new external row, using the result of evaluating the specified expressions as content.

    children

    A list of expression to use as content of the external row.

  50. case class CreateNamedStruct(children: Seq[Expression]) extends Expression with Product with Serializable

    Permalink

    Creates a struct with the given field names and values

    Creates a struct with the given field names and values

    children

    Seq(name1, val1, name2, val2, ...)

  51. case class CreateNamedStructUnsafe(children: Seq[Expression]) extends Expression with Product with Serializable

    Permalink

    Creates a struct with the given field names and values.

    Creates a struct with the given field names and values. This is a variant that returns UnsafeRow directly. The unsafe projection operator replaces CreateStruct with this expression automatically at runtime.

    children

    Seq(name1, val1, name2, val2, ...)

  52. case class CreateStruct(children: Seq[Expression]) extends Expression with Product with Serializable

    Permalink

    Returns a Row containing the evaluation of all children expressions.

  53. case class CreateStructUnsafe(children: Seq[Expression]) extends Expression with Product with Serializable

    Permalink

    Returns a Row containing the evaluation of all children expressions.

    Returns a Row containing the evaluation of all children expressions. This is a variant that returns UnsafeRow directly. The unsafe projection operator replaces CreateStruct with this expression automatically at runtime.

  54. case class CurrentDate() extends LeafExpression with CodegenFallback with Product with Serializable

    Permalink

    Returns the current date at the start of query evaluation.

    Returns the current date at the start of query evaluation. All calls of current_date within the same query return the same value.

    There is no code generation since this expression should get constant folded by the optimizer.

  55. case class CurrentTimestamp() extends LeafExpression with CodegenFallback with Product with Serializable

    Permalink

    Returns the current timestamp at the start of query evaluation.

    Returns the current timestamp at the start of query evaluation. All calls of current_timestamp within the same query return the same value.

    There is no code generation since this expression should get constant folded by the optimizer.

  56. case class DateAdd(startDate: Expression, days: Expression) extends BinaryExpression with ImplicitCastInputTypes with Product with Serializable

    Permalink

    Adds a number of days to startdate.

  57. case class DateDiff(endDate: Expression, startDate: Expression) extends BinaryExpression with ImplicitCastInputTypes with Product with Serializable

    Permalink

    Returns the number of days from startDate to endDate.

  58. case class DateFormatClass(left: Expression, right: Expression) extends BinaryExpression with ImplicitCastInputTypes with Product with Serializable

    Permalink
  59. case class DateSub(startDate: Expression, days: Expression) extends BinaryExpression with ImplicitCastInputTypes with Product with Serializable

    Permalink

    Subtracts a number of days to startdate.

  60. case class DayOfMonth(child: Expression) extends UnaryExpression with ImplicitCastInputTypes with Product with Serializable

    Permalink
  61. case class DayOfYear(child: Expression) extends UnaryExpression with ImplicitCastInputTypes with Product with Serializable

    Permalink
  62. case class Decode(bin: Expression, charset: Expression) extends BinaryExpression with ImplicitCastInputTypes with Product with Serializable

    Permalink

    Decodes the first argument into a String using the provided character set (one of 'US-ASCII', 'ISO-8859-1', 'UTF-8', 'UTF-16BE', 'UTF-16LE', 'UTF-16').

    Decodes the first argument into a String using the provided character set (one of 'US-ASCII', 'ISO-8859-1', 'UTF-8', 'UTF-16BE', 'UTF-16LE', 'UTF-16'). If either argument is null, the result will also be null.

  63. case class DecodeUsingSerializer[T](child: Expression, tag: ClassTag[T], kryo: Boolean) extends UnaryExpression with Product with Serializable

    Permalink

    Serializes an input object using a generic serializer (Kryo or Java).

    Serializes an input object using a generic serializer (Kryo or Java). Note that the ClassTag is not an implicit parameter because TreeNode cannot copy implicit parameters.

    kryo

    if true, use Kryo. Otherwise, use Java.

  64. case class Divide(left: Expression, right: Expression) extends BinaryArithmetic with Product with Serializable

    Permalink
  65. case class Encode(value: Expression, charset: Expression) extends BinaryExpression with ImplicitCastInputTypes with Product with Serializable

    Permalink

    Encodes the first argument into a BINARY using the provided character set (one of 'US-ASCII', 'ISO-8859-1', 'UTF-8', 'UTF-16BE', 'UTF-16LE', 'UTF-16').

    Encodes the first argument into a BINARY using the provided character set (one of 'US-ASCII', 'ISO-8859-1', 'UTF-8', 'UTF-16BE', 'UTF-16LE', 'UTF-16'). If either argument is null, the result will also be null.

  66. case class EncodeUsingSerializer(child: Expression, kryo: Boolean) extends UnaryExpression with Product with Serializable

    Permalink

    Serializes an input object using a generic serializer (Kryo or Java).

    Serializes an input object using a generic serializer (Kryo or Java).

    kryo

    if true, use Kryo. Otherwise, use Java.

  67. case class EndsWith(left: Expression, right: Expression) extends BinaryExpression with StringPredicate with Product with Serializable

    Permalink

    A function that returns true if the string left ends with the string right.

  68. case class EqualNullSafe(left: Expression, right: Expression) extends BinaryComparison with Product with Serializable

    Permalink
  69. case class EqualTo(left: Expression, right: Expression) extends BinaryComparison with Product with Serializable

    Permalink
  70. class EquivalentExpressions extends AnyRef

    Permalink

    This class is used to compute equality of (sub)expression trees.

    This class is used to compute equality of (sub)expression trees. Expressions can be added to this class and they subsequently query for expression equality. Expression trees are considered equal if for the same input(s), the same result is produced.

  71. case class EulerNumber() extends LeafMathExpression with Product with Serializable

    Permalink

    Euler's number.

    Euler's number. Note that there is no code generation because this is only evaluated by the optimizer during constant folding.

  72. case class Exp(child: Expression) extends UnaryMathExpression with Product with Serializable

    Permalink
  73. trait ExpectsInputTypes extends Expression

    Permalink

    An trait that gets mixin to define the expected input types of an expression.

    An trait that gets mixin to define the expected input types of an expression.

    This trait is typically used by operator expressions (e.g. Add, Subtract) to define expected input types without any implicit casting.

    Most function expressions (e.g. Substring should extends ImplicitCastInputTypes) instead.

  74. case class Explode(child: Expression) extends UnaryExpression with Generator with CodegenFallback with Product with Serializable

    Permalink

    Given an input array produces a sequence of rows for each value in the array.

  75. case class Expm1(child: Expression) extends UnaryMathExpression with Product with Serializable

    Permalink
  76. case class ExprId(id: Long, jvmId: UUID) extends Product with Serializable

    Permalink

    A globally unique id for a given named expression.

    A globally unique id for a given named expression. Used to identify which attribute output by a relation is being referenced in a subsequent computation.

    The id field is unique within a given JVM, while the uuid is used to uniquely identify JVMs.

  77. abstract class Expression extends TreeNode[Expression]

    Permalink

    An expression in Catalyst.

    An expression in Catalyst.

    If an expression wants to be exposed in the function registry (so users can call it with "name(arguments...)", the concrete implementation must be a case class whose constructor arguments are all Expressions types. See Substring for an example.

    There are a few important traits:

    - Nondeterministic: an expression that is not deterministic. - Unevaluable: an expression that is not supposed to be evaluated. - CodegenFallback: an expression that does not have code gen implemented and falls back to interpreted mode.

    - LeafExpression: an expression that has no child. - UnaryExpression: an expression that has one child. - BinaryExpression: an expression that has two children. - BinaryOperator: a special case of BinaryExpression that requires two children to have the same output data type.

  78. class ExpressionDescription extends Annotation with Annotation with ClassfileAnnotation

    Permalink
  79. class ExpressionInfo extends AnyRef

    Permalink
  80. case class Factorial(child: Expression) extends UnaryExpression with ImplicitCastInputTypes with Product with Serializable

    Permalink
  81. case class FindInSet(left: Expression, right: Expression) extends BinaryExpression with ImplicitCastInputTypes with Product with Serializable

    Permalink

    A function that returns the index (1-based) of the given string (left) in the comma- delimited list (right).

    A function that returns the index (1-based) of the given string (left) in the comma- delimited list (right). Returns 0, if the string wasn't found or if the given string (left) contains a comma.

  82. case class Floor(child: Expression) extends UnaryMathExpression with Product with Serializable

    Permalink
  83. case class FormatNumber(x: Expression, d: Expression) extends BinaryExpression with ExpectsInputTypes with Product with Serializable

    Permalink

    Formats the number X to a format like '#,###,###.##', rounded to D decimal places, and returns the result as a string.

    Formats the number X to a format like '#,###,###.##', rounded to D decimal places, and returns the result as a string. If D is 0, the result has no decimal point or fractional part.

  84. case class FormatString(children: Expression*) extends Expression with ImplicitCastInputTypes with Product with Serializable

    Permalink

    Returns the input formatted according do printf-style format strings

  85. sealed trait FrameBoundary extends AnyRef

    Permalink

    The trait used to represent the type of a Window Frame Boundary.

  86. sealed trait FrameType extends AnyRef

    Permalink

    The trait used to represent the type of a Window Frame.

  87. case class FromUTCTimestamp(left: Expression, right: Expression) extends BinaryExpression with ImplicitCastInputTypes with Product with Serializable

    Permalink

    Assumes given timestamp is UTC and converts to given timezone.

  88. case class FromUnixTime(sec: Expression, format: Expression) extends BinaryExpression with ImplicitCastInputTypes with Product with Serializable

    Permalink

    Converts the number of seconds from unix epoch (1970-01-01 00:00:00 UTC) to a string representing the timestamp of that moment in the current system time zone in the given format.

    Converts the number of seconds from unix epoch (1970-01-01 00:00:00 UTC) to a string representing the timestamp of that moment in the current system time zone in the given format. If the format is missing, using format like "1970-01-01 00:00:00". Note that hive Language Manual says it returns 0 if fail, but in fact it returns null.

  89. trait Generator extends Expression

    Permalink

    An expression that produces zero or more rows given a single input row.

    An expression that produces zero or more rows given a single input row.

    Generators produce multiple output rows instead of a single value like other expressions, and thus they must have a schema to associate with the rows that are output.

    However, unlike row producing relational operators, which are either leaves or determine their output schema functionally from their input, generators can contain other expressions that might result in their modification by rules. This structure means that they might be copied multiple times after first determining their output schema. If a new output schema is created for each copy references up the tree might be rendered invalid. As a result generators must instead define a function makeOutput which is called only once when the schema is first requested. The attributes produced by this function will be automatically copied anytime rules result in changes to the Generator or its children.

  90. class GenericInternalRow extends InternalRow with BaseGenericInternalRow

    Permalink

    A internal row implementation that uses an array of objects as the underlying storage.

    A internal row implementation that uses an array of objects as the underlying storage. Note that, while the array is not copied, and thus could technically be mutated after creation, this is not allowed.

  91. class GenericInternalRowWithSchema extends GenericInternalRow

    Permalink

    This is used for serialization of Python DataFrame

  92. class GenericMutableRow extends MutableRow with BaseGenericInternalRow

    Permalink
  93. class GenericRow extends Row

    Permalink

    A row implementation that uses an array of objects as the underlying storage.

    A row implementation that uses an array of objects as the underlying storage. Note that, while the array is not copied, and thus could technically be mutated after creation, this is not allowed.

  94. class GenericRowWithSchema extends GenericRow

    Permalink
  95. case class GetArrayItem(child: Expression, ordinal: Expression) extends BinaryExpression with ExpectsInputTypes with Product with Serializable

    Permalink

    Returns the field at ordinal in the Array child.

    Returns the field at ordinal in the Array child.

    We need to do type checking here as ordinal expression maybe unresolved.

  96. case class GetArrayStructFields(child: Expression, field: StructField, ordinal: Int, numFields: Int, containsNull: Boolean) extends UnaryExpression with Product with Serializable

    Permalink

    Returns the array of value of fields in the Array of Struct child.

    Returns the array of value of fields in the Array of Struct child.

    No need to do type checking since it is handled by ExtractValue.

  97. case class GetJsonObject(json: Expression, path: Expression) extends BinaryExpression with ExpectsInputTypes with CodegenFallback with Product with Serializable

    Permalink

    Extracts json object from a json string based on json path specified, and returns json string of the extracted json object.

    Extracts json object from a json string based on json path specified, and returns json string of the extracted json object. It will return null if the input json string is invalid.

  98. case class GetMapValue(child: Expression, key: Expression) extends BinaryExpression with ExpectsInputTypes with Product with Serializable

    Permalink

    Returns the value of key key in Map child.

    Returns the value of key key in Map child.

    We need to do type checking here as key expression maybe unresolved.

  99. case class GetStructField(child: Expression, ordinal: Int, name: Option[String] = None) extends UnaryExpression with Product with Serializable

    Permalink

    Returns the value of fields in the Struct child.

    Returns the value of fields in the Struct child.

    No need to do type checking since it is handled by ExtractValue.

    Note that we can pass in the field name directly to keep case preserving in toString. For example, when get field yEAr from <year: int, month: int>, we should pass in yEAr.

  100. case class GreaterThan(left: Expression, right: Expression) extends BinaryComparison with Product with Serializable

    Permalink
  101. case class GreaterThanOrEqual(left: Expression, right: Expression) extends BinaryComparison with Product with Serializable

    Permalink
  102. case class Greatest(children: Seq[Expression]) extends Expression with Product with Serializable

    Permalink

    A function that returns the greatest value of all parameters, skipping null values.

    A function that returns the greatest value of all parameters, skipping null values. It takes at least 2 parameters, and returns null iff all parameters are null.

  103. case class Hex(child: Expression) extends UnaryExpression with ImplicitCastInputTypes with Product with Serializable

    Permalink

    If the argument is an INT or binary, hex returns the number as a STRING in hexadecimal format.

    If the argument is an INT or binary, hex returns the number as a STRING in hexadecimal format. Otherwise if the number is a STRING, it converts each character into its hex representation and returns the resulting STRING. Negative numbers would be treated as two's complement.

  104. case class Hour(child: Expression) extends UnaryExpression with ImplicitCastInputTypes with Product with Serializable

    Permalink
  105. case class Hypot(left: Expression, right: Expression) extends BinaryMathExpression with Product with Serializable

    Permalink
  106. case class If(predicate: Expression, trueValue: Expression, falseValue: Expression) extends Expression with Product with Serializable

    Permalink
  107. trait ImplicitCastInputTypes extends Expression with ExpectsInputTypes

    Permalink

    A mixin for the analyzer to perform implicit type casting using ImplicitTypeCasts.

  108. case class In(value: Expression, list: Seq[Expression]) extends Expression with Predicate with ImplicitCastInputTypes with Product with Serializable

    Permalink

    Evaluates to true if list contains value.

  109. case class InSet(child: Expression, hset: Set[Any]) extends UnaryExpression with Predicate with Product with Serializable

    Permalink

    Optimized version of In clause, when all filter values of In clause are static.

  110. case class InitCap(child: Expression) extends UnaryExpression with ImplicitCastInputTypes with Product with Serializable

    Permalink

    Returns string, with the first letter of each word in uppercase.

    Returns string, with the first letter of each word in uppercase. Words are delimited by whitespace.

  111. case class InitializeJavaBean(beanInstance: Expression, setters: Map[String, Expression]) extends Expression with Product with Serializable

    Permalink

    Initialize a Java Bean instance by setting its field values via setters.

  112. case class InputFileName() extends LeafExpression with Nondeterministic with Product with Serializable

    Permalink

    Expression that returns the name of the current file being read in using SqlNewHadoopRDD

  113. case class InterpretedMutableProjection(expressions: Seq[Expression]) extends MutableProjection with Product with Serializable

    Permalink

    A MutableProjection that is calculated by calling eval on each of the specified expressions.

    A MutableProjection that is calculated by calling eval on each of the specified expressions.

    expressions

    a sequence of expressions that determine the value of each column of the output row.

  114. class InterpretedOrdering extends Ordering[InternalRow]

    Permalink

    An interpreted row ordering comparator.

  115. class InterpretedProjection extends Projection

    Permalink

    A Projection that is calculated by calling the eval of each of the specified expressions.

  116. case class Invoke(targetObject: Expression, functionName: String, dataType: DataType, arguments: Seq[Expression] = Nil) extends Expression with Product with Serializable

    Permalink

    Calls the specified function on an object, optionally passing arguments.

    Calls the specified function on an object, optionally passing arguments. If the targetObject expression evaluates to null then null will be returned.

    In some cases, due to erasure, the schema may expect a primitive type when in fact the method is returning java.lang.Object. In this case, we will generate code that attempts to unbox the value automatically.

    targetObject

    An expression that will return the object to call the method on.

    functionName

    The name of the method to call.

    dataType

    The expected return type of the function.

    arguments

    An optional list of expressions, whos evaluation will be passed to the function.

  117. case class IsNaN(child: Expression) extends UnaryExpression with Predicate with ImplicitCastInputTypes with Product with Serializable

    Permalink

    Evaluates to true iff it's NaN.

  118. case class IsNotNull(child: Expression) extends UnaryExpression with Predicate with Product with Serializable

    Permalink

    An expression that is evaluated to true if the input is not null.

  119. case class IsNull(child: Expression) extends UnaryExpression with Predicate with Product with Serializable

    Permalink

    An expression that is evaluated to true if the input is null.

  120. class JoinedRow extends InternalRow

    Permalink

    A mutable wrapper that makes two rows appear as a single concatenated row.

    A mutable wrapper that makes two rows appear as a single concatenated row. Designed to be instantiated once per thread and reused.

  121. case class JsonTuple(children: Seq[Expression]) extends Expression with Generator with CodegenFallback with Product with Serializable

    Permalink
  122. case class LambdaVariable(value: String, isNull: String, dataType: DataType) extends LeafExpression with Unevaluable with Product with Serializable

    Permalink

    A place holder for the loop variable used in MapObjects.

    A place holder for the loop variable used in MapObjects. This should never be constructed manually, but will instead be passed into the provided lambda function.

  123. case class LastDay(startDate: Expression) extends UnaryExpression with ImplicitCastInputTypes with Product with Serializable

    Permalink

    Returns the last day of the month which the date belongs to.

  124. abstract class LeafExpression extends Expression

    Permalink

    A leaf expression, i.e.

    A leaf expression, i.e. one without any child expressions.

  125. abstract class LeafMathExpression extends LeafExpression with CodegenFallback

    Permalink

    A leaf expression specifically for math constants.

    A leaf expression specifically for math constants. Math constants expect no input.

    There is no code generation because they should get constant folded by the optimizer.

  126. case class Least(children: Seq[Expression]) extends Expression with Product with Serializable

    Permalink

    A function that returns the least value of all parameters, skipping null values.

    A function that returns the least value of all parameters, skipping null values. It takes at least 2 parameters, and returns null iff all parameters are null.

  127. case class Length(child: Expression) extends UnaryExpression with ExpectsInputTypes with Product with Serializable

    Permalink

    A function that return the length of the given string or binary expression.

  128. case class LessThan(left: Expression, right: Expression) extends BinaryComparison with Product with Serializable

    Permalink
  129. case class LessThanOrEqual(left: Expression, right: Expression) extends BinaryComparison with Product with Serializable

    Permalink
  130. case class Levenshtein(left: Expression, right: Expression) extends BinaryExpression with ImplicitCastInputTypes with Product with Serializable

    Permalink

    A function that return the Levenshtein distance between the two given strings.

  131. case class Like(left: Expression, right: Expression) extends BinaryExpression with StringRegexExpression with CodegenFallback with Product with Serializable

    Permalink

    Simple RegEx pattern matching function

  132. case class Literal(value: Any, dataType: DataType) extends LeafExpression with CodegenFallback with Product with Serializable

    Permalink

    In order to do type checking, use Literal.create() instead of constructor

  133. case class Log(child: Expression) extends UnaryLogExpression with Product with Serializable

    Permalink
  134. case class Log10(child: Expression) extends UnaryLogExpression with Product with Serializable

    Permalink
  135. case class Log1p(child: Expression) extends UnaryLogExpression with Product with Serializable

    Permalink
  136. case class Log2(child: Expression) extends UnaryLogExpression with Product with Serializable

    Permalink
  137. case class Logarithm(left: Expression, right: Expression) extends BinaryMathExpression with Product with Serializable

    Permalink

    Computes the logarithm of a number.

    Computes the logarithm of a number.

    left

    the logarithm base, default to e.

    right

    the number to compute the logarithm of.

  138. case class Lower(child: Expression) extends UnaryExpression with String2StringExpression with Product with Serializable

    Permalink

    A function that converts the characters of a string to lowercase.

    A function that converts the characters of a string to lowercase.

    Annotations
    @ExpressionDescription()
  139. case class MakeDecimal(child: Expression, precision: Int, scale: Int) extends UnaryExpression with Product with Serializable

    Permalink

    Create a Decimal from an unscaled Long value.

    Create a Decimal from an unscaled Long value. Note: this expression is internal and created only by the optimizer, we don't need to do type check for it.

  140. case class MapObjects extends Expression with Product with Serializable

    Permalink

    Applies the given expression to every element of a collection of items, returning the result as an ArrayType.

    Applies the given expression to every element of a collection of items, returning the result as an ArrayType. This is similar to a typical map operation, but where the lambda function is expressed using catalyst expressions.

    The following collection ObjectTypes are currently supported: Seq, Array, ArrayData, java.util.List

  141. case class MaxOf(left: Expression, right: Expression) extends BinaryArithmetic with Product with Serializable

    Permalink
  142. case class Md5(child: Expression) extends UnaryExpression with ImplicitCastInputTypes with Product with Serializable

    Permalink

    A function that calculates an MD5 128-bit checksum and returns it as a hex string For input of type BinaryType

  143. case class MinOf(left: Expression, right: Expression) extends BinaryArithmetic with Product with Serializable

    Permalink
  144. case class Minute(child: Expression) extends UnaryExpression with ImplicitCastInputTypes with Product with Serializable

    Permalink
  145. case class Month(child: Expression) extends UnaryExpression with ImplicitCastInputTypes with Product with Serializable

    Permalink
  146. case class MonthsBetween(date1: Expression, date2: Expression) extends BinaryExpression with ImplicitCastInputTypes with Product with Serializable

    Permalink

    Returns number of months between dates date1 and date2.

  147. case class Multiply(left: Expression, right: Expression) extends BinaryArithmetic with Product with Serializable

    Permalink
  148. final class MutableAny extends MutableValue

    Permalink
  149. final class MutableBoolean extends MutableValue

    Permalink
  150. final class MutableByte extends MutableValue

    Permalink
  151. final class MutableDouble extends MutableValue

    Permalink
  152. final class MutableFloat extends MutableValue

    Permalink
  153. final class MutableInt extends MutableValue

    Permalink
  154. case class MutableLiteral(value: Any, dataType: DataType, nullable: Boolean = true) extends LeafExpression with CodegenFallback with Product with Serializable

    Permalink
  155. final class MutableLong extends MutableValue

    Permalink
  156. abstract class MutableProjection extends Projection

    Permalink

    Converts a InternalRow to another Row given a sequence of expression that define each column of the new row.

    Converts a InternalRow to another Row given a sequence of expression that define each column of the new row. If the schema of the input row is specified, then the given expression will be bound to that schema.

    In contrast to a normal projection, a MutableProjection reuses the same underlying row object each time an input row is added. This significantly reduces the cost of calculating the projection, but means that it is not safe to hold on to a reference to a InternalRow after next() has been called on the Iterator that produced it. Instead, the user must call InternalRow.copy() and hold on to the returned InternalRow before calling next().

  157. abstract class MutableRow extends InternalRow

    Permalink

    An extended interface to InternalRow that allows the values for each column to be updated.

    An extended interface to InternalRow that allows the values for each column to be updated. Setting a value through a primitive function implicitly marks that column as not null.

  158. final class MutableShort extends MutableValue

    Permalink
  159. abstract class MutableValue extends Serializable

    Permalink

    A parent class for mutable container objects that are reused when the values are changed, resulting in less garbage.

    A parent class for mutable container objects that are reused when the values are changed, resulting in less garbage. These values are held by a SpecificMutableRow.

    The following code was roughly used to generate these objects:

    val types = "Int,Float,Boolean,Double,Short,Long,Byte,Any".split(",")
    types.map {tpe =>
    s"""
    final class Mutable$tpe extends MutableValue {
      var value: $tpe = 0
      def boxed = if (isNull) null else value
      def update(v: Any) = value = {
        isNull = false
        v.asInstanceOf[$tpe]
      }
      def copy() = {
        val newCopy = new Mutable$tpe
        newCopy.isNull = isNull
        newCopy.value = value
        newCopy
      }
    }"""
    }.foreach(println)
    
    types.map { tpe =>
    s"""
      override def set$tpe(ordinal: Int, value: $tpe): Unit = {
        val currentValue = values(ordinal).asInstanceOf[Mutable$tpe]
        currentValue.isNull = false
        currentValue.value = value
      }
    
      override def get$tpe(i: Int): $tpe = {
        values(i).asInstanceOf[Mutable$tpe].value
      }"""
    }.foreach(println)
  160. case class NaNvl(left: Expression, right: Expression) extends BinaryExpression with ImplicitCastInputTypes with Product with Serializable

    Permalink

    An Expression evaluates to left iff it's not NaN, or evaluates to right otherwise.

    An Expression evaluates to left iff it's not NaN, or evaluates to right otherwise. This Expression is useful for mapping NaN values to null.

  161. trait NamedExpression extends Expression

    Permalink

    An Expression that is named.

  162. case class NewInstance(cls: Class[_], arguments: Seq[Expression], propagateNull: Boolean, dataType: DataType, outerPointer: Option[Literal]) extends Expression with Product with Serializable

    Permalink

    Constructs a new instance of the given class, using the result of evaluating the specified expressions as arguments.

    Constructs a new instance of the given class, using the result of evaluating the specified expressions as arguments.

    cls

    The class to construct.

    arguments

    A list of expression to use as arguments to the constructor.

    propagateNull

    When true, if any of the arguments is null, then null will be returned instead of trying to construct the object.

    dataType

    The type of object being constructed, as a Spark SQL datatype. This allows you to manually specify the type when the object in question is a valid internal representation (i.e. ArrayData) instead of an object.

    outerPointer

    If the object being constructed is an inner class the outerPointer must for the containing class must be specified.

  163. case class NextDay(startDate: Expression, dayOfWeek: Expression) extends BinaryExpression with ImplicitCastInputTypes with Product with Serializable

    Permalink

    Returns the first date which is later than startDate and named as dayOfWeek.

    Returns the first date which is later than startDate and named as dayOfWeek. For example, NextDay(2015-07-27, Sunday) would return 2015-08-02, which is the first Sunday later than 2015-07-27.

    Allowed "dayOfWeek" is defined in DateTimeUtils.getDayOfWeekFromString.

  164. trait Nondeterministic extends Expression

    Permalink

    An expression that is nondeterministic.

  165. case class Not(child: Expression) extends UnaryExpression with Predicate with ImplicitCastInputTypes with Product with Serializable

    Permalink
  166. case class Or(left: Expression, right: Expression) extends BinaryOperator with Predicate with Product with Serializable

    Permalink
  167. case class Pi() extends LeafMathExpression with Product with Serializable

    Permalink

    Pi.

    Pi. Note that there is no code generation because this is only evaluated by the optimizer during constant folding.

  168. case class Pmod(left: Expression, right: Expression) extends BinaryArithmetic with Product with Serializable

    Permalink
  169. case class Pow(left: Expression, right: Expression) extends BinaryMathExpression with Product with Serializable

    Permalink
  170. trait Predicate extends Expression

    Permalink

    An Expression that returns a boolean value.

  171. trait PredicateHelper extends AnyRef

    Permalink
  172. case class PrettyAttribute(name: String, dataType: DataType = NullType) extends Attribute with Unevaluable with Product with Serializable

    Permalink

    A place holder used when printing expressions without debugging information such as the expression id or the unresolved indicator.

  173. abstract class Projection extends (InternalRow) ⇒ InternalRow

    Permalink

    Converts a InternalRow to another Row given a sequence of expression that define each column of the new row.

    Converts a InternalRow to another Row given a sequence of expression that define each column of the new row. If the schema of the input row is specified, then the given expression will be bound to that schema.

  174. case class PromotePrecision(child: Expression) extends UnaryExpression with Product with Serializable

    Permalink

    An expression used to wrap the children when promote the precision of DecimalType to avoid promote multiple times.

  175. case class Quarter(child: Expression) extends UnaryExpression with ImplicitCastInputTypes with Product with Serializable

    Permalink
  176. abstract class RDG extends LeafExpression with Nondeterministic

    Permalink

    A Random distribution generating expression.

    A Random distribution generating expression. TODO: This can be made generic to generate any type of random distribution, or any type of StructType.

    Since this expression is stateful, it cannot be a case object.

  177. case class RLike(left: Expression, right: Expression) extends BinaryExpression with StringRegexExpression with CodegenFallback with Product with Serializable

    Permalink
  178. case class Rand(seed: Long) extends RDG with Product with Serializable

    Permalink

    Generate a random column with i.i.d.

    Generate a random column with i.i.d. uniformly distributed values in [0, 1).

  179. case class Randn(seed: Long) extends RDG with Product with Serializable

    Permalink

    Generate a random column with i.i.d.

    Generate a random column with i.i.d. gaussian random distribution.

  180. case class RegExpExtract(subject: Expression, regexp: Expression, idx: Expression) extends TernaryExpression with ImplicitCastInputTypes with Product with Serializable

    Permalink

    Extract a specific(idx) group identified by a Java regex.

    Extract a specific(idx) group identified by a Java regex.

    NOTE: this expression is not THREAD-SAFE, as it has some internal mutable status.

  181. case class RegExpReplace(subject: Expression, regexp: Expression, rep: Expression) extends TernaryExpression with ImplicitCastInputTypes with Product with Serializable

    Permalink

    Replace all substrings of str that match regexp with rep.

    Replace all substrings of str that match regexp with rep.

    NOTE: this expression is not THREAD-SAFE, as it has some internal mutable status.

  182. case class Remainder(left: Expression, right: Expression) extends BinaryArithmetic with Product with Serializable

    Permalink
  183. case class Rint(child: Expression) extends UnaryMathExpression with Product with Serializable

    Permalink
  184. case class Round(child: Expression, scale: Expression) extends BinaryExpression with ImplicitCastInputTypes with Product with Serializable

    Permalink

    Round the child's result to scale decimal place when scale >= 0 or round at integral part when scale < 0.

    Round the child's result to scale decimal place when scale >= 0 or round at integral part when scale < 0. For example, round(31.415, 2) = 31.42 and round(31.415, -1) = 30.

    Child of IntegralType would round to itself when scale >= 0. Child of FractionalType whose value is NaN or Infinite would always round to itself.

    Round's dataType would always equal to child's dataType except for DecimalType, which would lead scale decrease from the origin DecimalType.

    child

    expr to be round, all NumericType is allowed as Input

    scale

    new scale to be round to, this should be a constant int at runtime

  185. case class ScalaUDF(function: AnyRef, dataType: DataType, children: Seq[Expression], inputTypes: Seq[DataType] = Nil) extends Expression with ImplicitCastInputTypes with Product with Serializable

    Permalink

    User-defined function.

    User-defined function.

    function

    The user defined scala function to run. Note that if you use primitive parameters, you are not able to check if it is null or not, and the UDF will return null for you if the primitive input is null. Use boxed type or Option if you wanna do the null-handling yourself.

    dataType

    Return type of function.

    children

    The input expressions of this UDF.

    inputTypes

    The expected input types of this UDF.

  186. case class Second(child: Expression) extends UnaryExpression with ImplicitCastInputTypes with Product with Serializable

    Permalink
  187. case class Sha1(child: Expression) extends UnaryExpression with ImplicitCastInputTypes with Product with Serializable

    Permalink

    A function that calculates a sha1 hash value and returns it as a hex string For input of type BinaryType or StringType

  188. case class Sha2(left: Expression, right: Expression) extends BinaryExpression with Serializable with ImplicitCastInputTypes with Product

    Permalink

    A function that calculates the SHA-2 family of functions (SHA-224, SHA-256, SHA-384, and SHA-512) and returns it as a hex string.

    A function that calculates the SHA-2 family of functions (SHA-224, SHA-256, SHA-384, and SHA-512) and returns it as a hex string. The first argument is the string or binary to be hashed. The second argument indicates the desired bit length of the result, which must have a value of 224, 256, 384, 512, or 0 (which is equivalent to 256). SHA-224 is supported starting from Java 8. If asking for an unsupported SHA function, the return value is NULL. If either argument is NULL or the hash length is not one of the permitted values, the return value is NULL.

  189. case class ShiftLeft(left: Expression, right: Expression) extends BinaryExpression with ImplicitCastInputTypes with Product with Serializable

    Permalink

    Bitwise unsigned left shift.

    Bitwise unsigned left shift.

    left

    the base number to shift.

    right

    number of bits to left shift.

  190. case class ShiftRight(left: Expression, right: Expression) extends BinaryExpression with ImplicitCastInputTypes with Product with Serializable

    Permalink

    Bitwise unsigned left shift.

    Bitwise unsigned left shift.

    left

    the base number to shift.

    right

    number of bits to left shift.

  191. case class ShiftRightUnsigned(left: Expression, right: Expression) extends BinaryExpression with ImplicitCastInputTypes with Product with Serializable

    Permalink

    Bitwise unsigned right shift, for integer and long data type.

    Bitwise unsigned right shift, for integer and long data type.

    left

    the base number.

    right

    the number of bits to right shift.

  192. case class Signum(child: Expression) extends UnaryMathExpression with Product with Serializable

    Permalink
  193. case class Sin(child: Expression) extends UnaryMathExpression with Product with Serializable

    Permalink
  194. case class Sinh(child: Expression) extends UnaryMathExpression with Product with Serializable

    Permalink
  195. case class Size(child: Expression) extends UnaryExpression with ExpectsInputTypes with Product with Serializable

    Permalink

    Given an array or map, returns its size.

  196. case class SortArray(base: Expression, ascendingOrder: Expression) extends BinaryExpression with ExpectsInputTypes with CodegenFallback with Product with Serializable

    Permalink

    Sorts the input array in ascending / descending order according to the natural ordering of the array elements and returns it.

  197. sealed abstract class SortDirection extends AnyRef

    Permalink
  198. case class SortOrder(child: Expression, direction: SortDirection) extends UnaryExpression with Unevaluable with Product with Serializable

    Permalink

    An expression that can be used to sort a tuple.

    An expression that can be used to sort a tuple. This class extends expression primarily so that transformations over expression will descend into its child.

  199. case class SortPrefix(child: SortOrder) extends UnaryExpression with Product with Serializable

    Permalink

    An expression to generate a 64-bit long prefix used in sorting.

  200. case class SoundEx(child: Expression) extends UnaryExpression with ExpectsInputTypes with Product with Serializable

    Permalink

    A function that return soundex code of the given string expression.

  201. trait SpecializedGetters extends AnyRef

    Permalink
  202. final class SpecificMutableRow extends MutableRow with BaseGenericInternalRow

    Permalink

    A row type that holds an array specialized container objects, of type MutableValue, chosen based on the dataTypes of each column.

    A row type that holds an array specialized container objects, of type MutableValue, chosen based on the dataTypes of each column. The intent is to decrease garbage when modifying the values of primitive columns.

  203. case class SpecifiedWindowFrame(frameType: FrameType, frameStart: FrameBoundary, frameEnd: FrameBoundary) extends WindowFrame with Product with Serializable

    Permalink

    A specified Window Frame.

  204. case class Sqrt(child: Expression) extends UnaryMathExpression with Product with Serializable

    Permalink
  205. case class StartsWith(left: Expression, right: Expression) extends BinaryExpression with StringPredicate with Product with Serializable

    Permalink

    A function that returns true if the string left starts with the string right.

  206. case class StaticInvoke(staticObject: Class[_], dataType: DataType, functionName: String, arguments: Seq[Expression] = Nil, propagateNull: Boolean = true) extends Expression with Product with Serializable

    Permalink

    Invokes a static function, returning the result.

    Invokes a static function, returning the result. By default, any of the arguments being null will result in returning null instead of calling the function.

    staticObject

    The target of the static call. This can either be the object itself (methods defined on scala objects), or the class object (static methods defined in java).

    dataType

    The expected return type of the function call

    functionName

    The name of the method to call.

    arguments

    An optional list of expressions to pass as arguments to the function.

    propagateNull

    When true, and any of the arguments is null, null will be returned instead of calling the function.

  207. trait String2StringExpression extends Expression with ImplicitCastInputTypes

    Permalink
  208. case class StringInstr(str: Expression, substr: Expression) extends BinaryExpression with ImplicitCastInputTypes with Product with Serializable

    Permalink

    A function that returns the position of the first occurrence of substr in the given string.

    A function that returns the position of the first occurrence of substr in the given string. Returns null if either of the arguments are null and returns 0 if substr could not be found in str.

    NOTE: that this is not zero based, but 1-based index. The first character in str has index 1.

  209. case class StringLPad(str: Expression, len: Expression, pad: Expression) extends TernaryExpression with ImplicitCastInputTypes with Product with Serializable

    Permalink

    Returns str, left-padded with pad to a length of len.

  210. case class StringLocate(substr: Expression, str: Expression, start: Expression) extends TernaryExpression with ImplicitCastInputTypes with Product with Serializable

    Permalink

    A function that returns the position of the first occurrence of substr in given string after position pos.

  211. trait StringPredicate extends Expression with Predicate with ImplicitCastInputTypes

    Permalink

    A base trait for functions that compare two strings, returning a boolean.

  212. case class StringRPad(str: Expression, len: Expression, pad: Expression) extends TernaryExpression with ImplicitCastInputTypes with Product with Serializable

    Permalink

    Returns str, right-padded with pad to a length of len.

  213. trait StringRegexExpression extends Expression with ImplicitCastInputTypes

    Permalink
  214. case class StringRepeat(str: Expression, times: Expression) extends BinaryExpression with ImplicitCastInputTypes with Product with Serializable

    Permalink

    Returns the string which repeat the given string value n times.

  215. case class StringReverse(child: Expression) extends UnaryExpression with String2StringExpression with Product with Serializable

    Permalink

    Returns the reversed given string.

  216. case class StringSpace(child: Expression) extends UnaryExpression with ImplicitCastInputTypes with Product with Serializable

    Permalink

    Returns a n spaces string.

  217. case class StringSplit(str: Expression, pattern: Expression) extends BinaryExpression with ImplicitCastInputTypes with Product with Serializable

    Permalink

    Splits str around pat (pattern is a regular expression).

  218. case class StringTranslate(srcExpr: Expression, matchingExpr: Expression, replaceExpr: Expression) extends TernaryExpression with ImplicitCastInputTypes with Product with Serializable

    Permalink

    A function translate any character in the srcExpr by a character in replaceExpr.

    A function translate any character in the srcExpr by a character in replaceExpr. The characters in replaceExpr is corresponding to the characters in matchingExpr. The translate will happen when any character in the string matching with the character in the matchingExpr.

  219. case class StringTrim(child: Expression) extends UnaryExpression with String2StringExpression with Product with Serializable

    Permalink

    A function that trim the spaces from both ends for the specified string.

  220. case class StringTrimLeft(child: Expression) extends UnaryExpression with String2StringExpression with Product with Serializable

    Permalink

    A function that trim the spaces from left end for given string.

  221. case class StringTrimRight(child: Expression) extends UnaryExpression with String2StringExpression with Product with Serializable

    Permalink

    A function that trim the spaces from right end for given string.

  222. case class Substring(str: Expression, pos: Expression, len: Expression) extends TernaryExpression with ImplicitCastInputTypes with Product with Serializable

    Permalink

    A function that takes a substring of its first argument starting at a given position.

    A function that takes a substring of its first argument starting at a given position. Defined for String and Binary types.

  223. case class SubstringIndex(strExpr: Expression, delimExpr: Expression, countExpr: Expression) extends TernaryExpression with ImplicitCastInputTypes with Product with Serializable

    Permalink

    Returns the substring from string str before count occurrences of the delimiter delim.

    Returns the substring from string str before count occurrences of the delimiter delim. If count is positive, everything the left of the final delimiter (counting from left) is returned. If count is negative, every to the right of the final delimiter (counting from the right) is returned. substring_index performs a case-sensitive match when searching for delim.

  224. case class Subtract(left: Expression, right: Expression) extends BinaryArithmetic with Product with Serializable

    Permalink
  225. case class Tan(child: Expression) extends UnaryMathExpression with Product with Serializable

    Permalink
  226. case class Tanh(child: Expression) extends UnaryMathExpression with Product with Serializable

    Permalink
  227. abstract class TernaryExpression extends Expression

    Permalink

    An expression with three inputs and one output.

    An expression with three inputs and one output. The output is by default evaluated to null if any input is evaluated to null.

  228. case class TimeAdd(start: Expression, interval: Expression) extends BinaryExpression with ImplicitCastInputTypes with Product with Serializable

    Permalink

    Adds an interval to timestamp.

  229. case class TimeSub(start: Expression, interval: Expression) extends BinaryExpression with ImplicitCastInputTypes with Product with Serializable

    Permalink

    Subtracts an interval from timestamp.

  230. case class ToDate(child: Expression) extends UnaryExpression with ImplicitCastInputTypes with Product with Serializable

    Permalink

    Returns the date part of a timestamp or string.

  231. case class ToDegrees(child: Expression) extends UnaryMathExpression with Product with Serializable

    Permalink
  232. case class ToRadians(child: Expression) extends UnaryMathExpression with Product with Serializable

    Permalink
  233. case class ToUTCTimestamp(left: Expression, right: Expression) extends BinaryExpression with ImplicitCastInputTypes with Product with Serializable

    Permalink

    Assumes given timestamp is in given timezone and converts to UTC.

  234. case class ToUnixTimestamp(timeExp: Expression, format: Expression) extends UnixTime with Product with Serializable

    Permalink

    Converts time string with given pattern.

    Converts time string with given pattern. Deterministic version of UnixTimestamp, must have at least one parameter.

  235. case class TruncDate(date: Expression, format: Expression) extends BinaryExpression with ImplicitCastInputTypes with Product with Serializable

    Permalink

    Returns date truncated to the unit specified by the format.

  236. case class UnBase64(child: Expression) extends UnaryExpression with ImplicitCastInputTypes with Product with Serializable

    Permalink

    Converts the argument from a base 64 string to BINARY.

  237. abstract class UnaryExpression extends Expression

    Permalink

    An expression with one input and one output.

    An expression with one input and one output. The output is by default evaluated to null if the input is evaluated to null.

  238. abstract class UnaryLogExpression extends UnaryMathExpression

    Permalink
  239. abstract class UnaryMathExpression extends UnaryExpression with Serializable with ImplicitCastInputTypes

    Permalink

    A unary expression specifically for math functions.

    A unary expression specifically for math functions. Math Functions expect a specific type of input format, therefore these functions extend ExpectsInputTypes.

  240. case class UnaryMinus(child: Expression) extends UnaryExpression with ExpectsInputTypes with Product with Serializable

    Permalink
  241. case class UnaryPositive(child: Expression) extends UnaryExpression with ExpectsInputTypes with Product with Serializable

    Permalink
  242. trait Unevaluable extends Expression

    Permalink

    An expression that cannot be evaluated.

    An expression that cannot be evaluated. Some expressions don't live past analysis or optimization time (e.g. Star). This trait is used by those expressions.

  243. case class Unhex(child: Expression) extends UnaryExpression with ImplicitCastInputTypes with Product with Serializable

    Permalink

    Performs the inverse operation of HEX.

    Performs the inverse operation of HEX. Resulting characters are returned as a byte array.

  244. abstract class UnixTime extends BinaryExpression with ExpectsInputTypes

    Permalink
  245. case class UnixTimestamp(timeExp: Expression, format: Expression) extends UnixTime with Product with Serializable

    Permalink

    Converts time string with given pattern.

    Converts time string with given pattern. (see [http://docs.oracle.com/javase/tutorial/i18n/format/simpleDateFormat.html]) to Unix time stamp (in seconds), returns null if fail. Note that hive Language Manual says it returns 0 if fail, but in fact it returns null. If the second parameter is missing, use "yyyy-MM-dd HH:mm:ss". If no parameters provided, the first parameter will be current_timestamp. If the first parameter is a Date or Timestamp instead of String, we will ignore the second parameter.

  246. case class UnresolvedWindowExpression(child: UnresolvedWindowFunction, windowSpec: WindowSpecReference) extends UnaryExpression with Unevaluable with Product with Serializable

    Permalink
  247. case class UnresolvedWindowFunction(name: String, children: Seq[Expression]) extends Expression with WindowFunction with Unevaluable with Product with Serializable

    Permalink
  248. class UnsafeArrayData extends ArrayData

    Permalink
  249. class UnsafeMapData extends MapData

    Permalink
  250. abstract class UnsafeProjection extends Projection

    Permalink

    A projection that returns UnsafeRow.

  251. final class UnsafeRow extends MutableRow with Externalizable with KryoSerializable

    Permalink
  252. case class UnscaledValue(child: Expression) extends UnaryExpression with Product with Serializable

    Permalink

    Return the unscaled Long value of a Decimal, assuming it fits in a Long.

    Return the unscaled Long value of a Decimal, assuming it fits in a Long. Note: this expression is internal and created only by the optimizer, we don't need to do type check for it.

  253. case class UnwrapOption(dataType: DataType, child: Expression) extends UnaryExpression with ExpectsInputTypes with Product with Serializable

    Permalink

    Given an expression that returns on object of type Option[_], this expression unwraps the option into the specified Spark SQL datatype.

    Given an expression that returns on object of type Option[_], this expression unwraps the option into the specified Spark SQL datatype. In the case of None, the nullbit is set instead.

    dataType

    The expected unwrapped option type.

    child

    An expression that returns an Option

  254. case class UpCast(child: Expression, dataType: DataType, walkedTypePath: Seq[String]) extends UnaryExpression with Unevaluable with Product with Serializable

    Permalink

    Cast the child expression to the target data type, but will throw error if the cast might truncate, e.g.

    Cast the child expression to the target data type, but will throw error if the cast might truncate, e.g. long -> int, timestamp -> data.

  255. case class Upper(child: Expression) extends UnaryExpression with String2StringExpression with Product with Serializable

    Permalink

    A function that converts the characters of a string to uppercase.

    A function that converts the characters of a string to uppercase.

    Annotations
    @ExpressionDescription()
  256. case class UserDefinedGenerator(elementTypes: Seq[(DataType, Boolean, String)], function: (Row) ⇒ TraversableOnce[InternalRow], children: Seq[Expression]) extends Expression with Generator with CodegenFallback with Product with Serializable

    Permalink

    A generator that produces its output using the provided lambda function.

  257. case class ValueFollowing(value: Int) extends FrameBoundary with Product with Serializable

    Permalink

    <value> FOLLOWING boundary.

  258. case class ValuePreceding(value: Int) extends FrameBoundary with Product with Serializable

    Permalink

    <value> PRECEDING boundary.

  259. case class WeekOfYear(child: Expression) extends UnaryExpression with ImplicitCastInputTypes with Product with Serializable

    Permalink
  260. case class WindowExpression(windowFunction: WindowFunction, windowSpec: WindowSpecDefinition) extends Expression with Unevaluable with Product with Serializable

    Permalink
  261. sealed trait WindowFrame extends AnyRef

    Permalink

    The trait used to represent the a Window Frame.

  262. trait WindowFunction extends Expression

    Permalink

    Every window function needs to maintain a output buffer for its output.

    Every window function needs to maintain a output buffer for its output. It should expect that for a n-row window frame, it will be called n times to retrieve value corresponding with these n rows.

  263. sealed trait WindowSpec extends AnyRef

    Permalink

    The trait of the Window Specification (specified in the OVER clause or WINDOW clause) for Window Functions.

  264. case class WindowSpecDefinition(partitionSpec: Seq[Expression], orderSpec: Seq[SortOrder], frameSpecification: WindowFrame) extends Expression with WindowSpec with Unevaluable with Product with Serializable

    Permalink

    The specification for a window function.

    The specification for a window function.

    partitionSpec

    It defines the way that input rows are partitioned.

    orderSpec

    It defines the ordering of rows in a partition.

    frameSpecification

    It defines the window frame in a partition.

  265. case class WindowSpecReference(name: String) extends WindowSpec with Product with Serializable

    Permalink

    A Window specification reference that refers to the WindowSpecDefinition defined under the name name.

  266. case class WrapOption(child: Expression, optType: DataType) extends UnaryExpression with ExpectsInputTypes with Product with Serializable

    Permalink

    Converts the result of evaluating child into an option, checking both the isNull bit and (in the case of reference types) equality with null.

    Converts the result of evaluating child into an option, checking both the isNull bit and (in the case of reference types) equality with null.

    child

    The expression to evaluate and wrap.

    optType

    The type of this option.

  267. case class Year(child: Expression) extends UnaryExpression with ImplicitCastInputTypes with Product with Serializable

    Permalink

Value Members

  1. object Ascending extends SortDirection with Product with Serializable

    Permalink
  2. object AttributeMap extends Serializable

    Permalink

    Builds a map that is keyed by an Attribute's expression id.

    Builds a map that is keyed by an Attribute's expression id. Using the expression id allows values to be looked up even when the attributes used differ cosmetically (i.e., the capitalization of the name, or the expected nullability).

  3. object AttributeSet extends Serializable

    Permalink
  4. object BindReferences extends Logging

    Permalink
  5. object Cast extends Serializable

    Permalink
  6. object CurrentRow extends FrameBoundary with Product with Serializable

    Permalink

    CURRENT ROW boundary.

  7. object Descending extends SortDirection with Product with Serializable

    Permalink
  8. val EmptyRow: InternalRow

    Permalink

    Used as input into expressions whose output does not depend on any input value.

  9. object ExprId extends Serializable

    Permalink
  10. object ExtractValue

    Permalink
  11. object Factorial extends Serializable

    Permalink
  12. object FrameBoundaryExtractor

    Permalink

    Extractor for making working with frame boundaries easier.

  13. object FromUnsafeProjection

    Permalink

    A projection that could turn UnsafeRow into GenericInternalRow

  14. object Hex extends Serializable

    Permalink
  15. object IntegerLiteral

    Permalink

    Extractor for retrieving Int literals.

  16. object InterpretedOrdering extends Serializable

    Permalink
  17. object InterpretedPredicate

    Permalink
  18. object Literal extends Serializable

    Permalink
  19. object MapObjects extends Serializable

    Permalink
  20. object NamedExpression

    Permalink
  21. object NewInstance extends Serializable

    Permalink
  22. object NonNullLiteral

    Permalink

    An extractor that matches non-null literal values

  23. object RangeFrame extends FrameType with Product with Serializable

    Permalink

    RangeFrame treats rows in a partition as groups of peers.

    RangeFrame treats rows in a partition as groups of peers. All rows having the same ORDER BY ordering are considered as peers. When a ValuePreceding or a ValueFollowing is used as its FrameBoundary, the value is considered as a logical offset. For example, assuming the value of the current row's ORDER BY expression expr is v, RANGE BETWEEN 1 PRECEDING AND 1 FOLLOWING represents a frame containing rows whose values expr are in the range of [v-1, v+1].

    If ORDER BY clause is not defined, all rows in the partition is considered as peers of the current row.

  24. object RowFrame extends FrameType with Product with Serializable

    Permalink

    RowFrame treats rows in a partition individually.

    RowFrame treats rows in a partition individually. When a ValuePreceding or a ValueFollowing is used as its FrameBoundary, the value is considered as a physical offset. For example, ROW BETWEEN 1 PRECEDING AND 1 FOLLOWING represents a 3-row frame, from the row precedes the current row to the row follows the current row.

  25. object RowOrdering

    Permalink
  26. object SpecifiedWindowFrame extends Serializable

    Permalink
  27. object StringTranslate extends Serializable

    Permalink
  28. object UnboundedFollowing extends FrameBoundary with Product with Serializable

    Permalink

    UNBOUNDED FOLLOWING boundary.

  29. object UnboundedPreceding extends FrameBoundary with Product with Serializable

    Permalink

    UNBOUNDED PRECEDING boundary.

  30. object UnsafeProjection

    Permalink
  31. object UnspecifiedFrame extends WindowFrame with Product with Serializable

    Permalink

    Used as a place holder when a frame specification is not defined.

  32. object VirtualColumn

    Permalink
  33. package aggregate

    Permalink
  34. package codegen

    Permalink

    A collection of generators that build custom bytecode at runtime for performing the evaluation of catalyst expression.

Inherited from AnyRef

Inherited from Any

Ungrouped