expressions
A function that get the absolute value of the numeric value.
A function that get the absolute value of the numeric value.
Returns the date that is num_months after start_date.
Used to assign a new name to a computation.
Used to assign a new name to a computation. For example the SQL expression "1 + 1 AS a" could be represented as follows: Alias(Add(Literal(1), Literal(1)), "a")()
Note that exprId and qualifiers are in a separate parameter list because we only pattern match on child and name.
the computation being performed
the name to be associated with the result of computing child.
A globally unique id used to check if an AttributeReference refers to this alias. Auto-assigned if left blank.
Explicit metadata associated with this alias that overwrites child's.
Checks if the array (left) has the element (right)
Returns the numeric value of the first character of str.
Asserts that input values of a non-nullable child expression are not null.
Asserts that input values of a non-nullable child expression are not null.
Note that there are cases where child.nullable == true, while we still needs to add this
assertion. Consider a nullable column s whose data type is a struct containing a non-nullable
Int field named i. Expression s.i is nullable because s can be null. However, for all
non-null s, s.i can't be null.
A predicate that is evaluated to be true if there are at least n non-null and non-NaN values.
A reference to an attribute produced by another operator in the tree.
A reference to an attribute produced by another operator in the tree.
The name of this attribute, should only be used during analysis or for debugging.
The DataType of this attribute.
True if null is a valid value for this attribute.
The metadata of this attribute.
A globally unique id used to check if different AttributeReferences refer to the same attribute.
a list of strings that can be used to referred to this attribute in a fully qualified way. Consider the examples tableName.name, subQueryAlias.name. tableName and subQueryAlias are possible qualifiers.
Helper functions for working with Seq[Attribute].
A Set designed to hold AttributeReference objects, that performs equality checking using expression id instead of standard java equality.
A Set designed to hold AttributeReference objects, that performs equality checking using expression id instead of standard java equality. Using expression id means that these sets will correctly test for membership, even when the AttributeReferences in question differ cosmetically (e.g., the names have different capitalizations).
Note that we do not override equality for Attribute references as it is really weird when
AttributeReference("a"...) == AttrributeReference("b", ...). This tactic leads to broken tests,
and also makes doing transformations hard (we always try keep older trees instead of new ones
when the transformation was a no-op).
Converts the argument from binary to a base 64 string.
An extended version of InternalRow that implements all special getters, toString
and equals/hashCode by genericGet.
An expression with two inputs and one output.
An expression with two inputs and one output. The output is by default evaluated to null if any input is evaluated to null.
A binary expression specifically for math functions that take two Doubles as input and returns
a Double.
A BinaryExpression that is an operator, with two properties:
A BinaryExpression that is an operator, with two properties:
1. The string representation is "x symbol y", rather than "funcName(x, y)". 2. Two inputs are expected to the be same type. If the two inputs have different types, the analyzer will find the tightest common type and do the proper type casting.
A function that calculates bitwise and(&) of two numbers.
A function that calculates bitwise and(&) of two numbers.
Code generation inherited from BinaryArithmetic.
A function that calculates bitwise not(~) of a number.
A function that calculates bitwise or(|) of two numbers.
A function that calculates bitwise or(|) of two numbers.
Code generation inherited from BinaryArithmetic.
A function that calculates bitwise xor of two numbers.
A function that calculates bitwise xor of two numbers.
Code generation inherited from BinaryArithmetic.
A bound reference points to a specific slot in the input tuple, allowing the actual value to be retrieved more efficiently.
A bound reference points to a specific slot in the input tuple, allowing the actual value to be retrieved more efficiently. However, since operations like column pruning can change the layout of intermediate tuples, BindReferences should be run after all such transformations.
Case statements of the form "CASE a WHEN b THEN c [WHEN d THEN e]* [ELSE f] END".
Case statements of the form "CASE a WHEN b THEN c [WHEN d THEN e]* [ELSE f] END". Refer to this link for the corresponding semantics: https://cwiki.apache.org/confluence/display/Hive/LanguageManual+UDF#LanguageManualUDF-ConditionalFunctions
Case statements of the form "CASE WHEN a THEN b [WHEN c THEN d]* [ELSE e] END".
Case statements of the form "CASE WHEN a THEN b [WHEN c THEN d]* [ELSE e] END". Refer to this link for the corresponding semantics: https://cwiki.apache.org/confluence/display/Hive/LanguageManual+UDF#LanguageManualUDF-ConditionalFunctions
Cast the child expression to the target data type.
Rounds the decimal to given scale and check whether the decimal can fit in provided precision or not, returns null if not.
An expression that is evaluated to the first non-null input.
An expression that is evaluated to the first non-null input.
coalesce(1, 2) => 1 coalesce(null, 1, 2) => 1 coalesce(null, null, 2) => 2 coalesce(null, null, null) => null
An expression that concatenates multiple input strings into a single string.
An expression that concatenates multiple input strings into a single string. If any input is null, concat returns null.
An expression that concatenates multiple input strings or array of strings into a single string, using a given separator (the first child).
An expression that concatenates multiple input strings or array of strings into a single string, using a given separator (the first child).
Returns null if the separator is null. Otherwise, concat_ws skips all null values.
A function that returns true if the string left contains the string right.
Convert a num from one base to another
Convert a num from one base to another
the number to be converted
from which base
to which base
A function that computes a cyclic redundancy check value and returns it as a bigint For input of type BinaryType
Returns an Array containing the evaluation of all children expressions.
Constructs a new external row, using the result of evaluating the specified expressions as content.
Constructs a new external row, using the result of evaluating the specified expressions as content.
A list of expression to use as content of the external row.
Creates a struct with the given field names and values
Creates a struct with the given field names and values
Seq(name1, val1, name2, val2, ...)
Creates a struct with the given field names and values.
Creates a struct with the given field names and values. This is a variant that returns UnsafeRow directly. The unsafe projection operator replaces CreateStruct with this expression automatically at runtime.
Seq(name1, val1, name2, val2, ...)
Returns a Row containing the evaluation of all children expressions.
Returns a Row containing the evaluation of all children expressions.
Returns a Row containing the evaluation of all children expressions. This is a variant that returns UnsafeRow directly. The unsafe projection operator replaces CreateStruct with this expression automatically at runtime.
Returns the current date at the start of query evaluation.
Returns the current date at the start of query evaluation. All calls of current_date within the same query return the same value.
There is no code generation since this expression should get constant folded by the optimizer.
Returns the current timestamp at the start of query evaluation.
Returns the current timestamp at the start of query evaluation. All calls of current_timestamp within the same query return the same value.
There is no code generation since this expression should get constant folded by the optimizer.
Adds a number of days to startdate.
Returns the number of days from startDate to endDate.
Subtracts a number of days to startdate.
Decodes the first argument into a String using the provided character set (one of 'US-ASCII', 'ISO-8859-1', 'UTF-8', 'UTF-16BE', 'UTF-16LE', 'UTF-16').
Decodes the first argument into a String using the provided character set (one of 'US-ASCII', 'ISO-8859-1', 'UTF-8', 'UTF-16BE', 'UTF-16LE', 'UTF-16'). If either argument is null, the result will also be null.
Serializes an input object using a generic serializer (Kryo or Java).
Serializes an input object using a generic serializer (Kryo or Java). Note that the ClassTag is not an implicit parameter because TreeNode cannot copy implicit parameters.
if true, use Kryo. Otherwise, use Java.
Encodes the first argument into a BINARY using the provided character set (one of 'US-ASCII', 'ISO-8859-1', 'UTF-8', 'UTF-16BE', 'UTF-16LE', 'UTF-16').
Encodes the first argument into a BINARY using the provided character set (one of 'US-ASCII', 'ISO-8859-1', 'UTF-8', 'UTF-16BE', 'UTF-16LE', 'UTF-16'). If either argument is null, the result will also be null.
Serializes an input object using a generic serializer (Kryo or Java).
Serializes an input object using a generic serializer (Kryo or Java).
if true, use Kryo. Otherwise, use Java.
A function that returns true if the string left ends with the string right.
This class is used to compute equality of (sub)expression trees.
This class is used to compute equality of (sub)expression trees. Expressions can be added to this class and they subsequently query for expression equality. Expression trees are considered equal if for the same input(s), the same result is produced.
Euler's number.
Euler's number. Note that there is no code generation because this is only evaluated by the optimizer during constant folding.
An trait that gets mixin to define the expected input types of an expression.
An trait that gets mixin to define the expected input types of an expression.
This trait is typically used by operator expressions (e.g. Add, Subtract) to define expected input types without any implicit casting.
Most function expressions (e.g. Substring should extends ImplicitCastInputTypes) instead.
Given an input array produces a sequence of rows for each value in the array.
A globally unique id for a given named expression.
A globally unique id for a given named expression. Used to identify which attribute output by a relation is being referenced in a subsequent computation.
The id field is unique within a given JVM, while the uuid is used to uniquely identify JVMs.
An expression in Catalyst.
An expression in Catalyst.
If an expression wants to be exposed in the function registry (so users can call it with "name(arguments...)", the concrete implementation must be a case class whose constructor arguments are all Expressions types. See Substring for an example.
There are a few important traits:
- Nondeterministic: an expression that is not deterministic. - Unevaluable: an expression that is not supposed to be evaluated. - CodegenFallback: an expression that does not have code gen implemented and falls back to interpreted mode.
- LeafExpression: an expression that has no child. - UnaryExpression: an expression that has one child. - BinaryExpression: an expression that has two children. - BinaryOperator: a special case of BinaryExpression that requires two children to have the same output data type.
A function that returns the index (1-based) of the given string (left) in the comma- delimited list (right).
A function that returns the index (1-based) of the given string (left) in the comma- delimited list (right). Returns 0, if the string wasn't found or if the given string (left) contains a comma.
Formats the number X to a format like '#,###,###.##', rounded to D decimal places, and returns the result as a string.
Formats the number X to a format like '#,###,###.##', rounded to D decimal places, and returns the result as a string. If D is 0, the result has no decimal point or fractional part.
Returns the input formatted according do printf-style format strings
The trait used to represent the type of a Window Frame Boundary.
The trait used to represent the type of a Window Frame.
Assumes given timestamp is UTC and converts to given timezone.
Converts the number of seconds from unix epoch (1970-01-01 00:00:00 UTC) to a string representing the timestamp of that moment in the current system time zone in the given format.
Converts the number of seconds from unix epoch (1970-01-01 00:00:00 UTC) to a string representing the timestamp of that moment in the current system time zone in the given format. If the format is missing, using format like "1970-01-01 00:00:00". Note that hive Language Manual says it returns 0 if fail, but in fact it returns null.
An expression that produces zero or more rows given a single input row.
An expression that produces zero or more rows given a single input row.
Generators produce multiple output rows instead of a single value like other expressions, and thus they must have a schema to associate with the rows that are output.
However, unlike row producing relational operators, which are either leaves or determine their
output schema functionally from their input, generators can contain other expressions that
might result in their modification by rules. This structure means that they might be copied
multiple times after first determining their output schema. If a new output schema is created for
each copy references up the tree might be rendered invalid. As a result generators must
instead define a function makeOutput which is called only once when the schema is first
requested. The attributes produced by this function will be automatically copied anytime rules
result in changes to the Generator or its children.
A internal row implementation that uses an array of objects as the underlying storage.
A internal row implementation that uses an array of objects as the underlying storage. Note that, while the array is not copied, and thus could technically be mutated after creation, this is not allowed.
This is used for serialization of Python DataFrame
A row implementation that uses an array of objects as the underlying storage.
A row implementation that uses an array of objects as the underlying storage. Note that, while the array is not copied, and thus could technically be mutated after creation, this is not allowed.
Returns the field at ordinal in the Array child.
Returns the field at ordinal in the Array child.
We need to do type checking here as ordinal expression maybe unresolved.
Returns the array of value of fields in the Array of Struct child.
Returns the array of value of fields in the Array of Struct child.
No need to do type checking since it is handled by ExtractValue.
Extracts json object from a json string based on json path specified, and returns json string of the extracted json object.
Extracts json object from a json string based on json path specified, and returns json string of the extracted json object. It will return null if the input json string is invalid.
Returns the value of key key in Map child.
Returns the value of key key in Map child.
We need to do type checking here as key expression maybe unresolved.
Returns the value of fields in the Struct child.
Returns the value of fields in the Struct child.
No need to do type checking since it is handled by ExtractValue.
Note that we can pass in the field name directly to keep case preserving in toString.
For example, when get field yEAr from <year: int, month: int>, we should pass in yEAr.
A function that returns the greatest value of all parameters, skipping null values.
A function that returns the greatest value of all parameters, skipping null values. It takes at least 2 parameters, and returns null iff all parameters are null.
If the argument is an INT or binary, hex returns the number as a STRING in hexadecimal format.
If the argument is an INT or binary, hex returns the number as a STRING in hexadecimal format. Otherwise if the number is a STRING, it converts each character into its hex representation and returns the resulting STRING. Negative numbers would be treated as two's complement.
A mixin for the analyzer to perform implicit type casting using ImplicitTypeCasts.
Evaluates to true if list contains value.
Optimized version of In clause, when all filter values of In clause are static.
Returns string, with the first letter of each word in uppercase.
Returns string, with the first letter of each word in uppercase. Words are delimited by whitespace.
Initialize a Java Bean instance by setting its field values via setters.
Expression that returns the name of the current file being read in using SqlNewHadoopRDD
A MutableProjection that is calculated by calling eval on each of the specified
expressions.
A MutableProjection that is calculated by calling eval on each of the specified
expressions.
a sequence of expressions that determine the value of each column of the output row.
An interpreted row ordering comparator.
A Projection that is calculated by calling the eval of each of the specified expressions.
Calls the specified function on an object, optionally passing arguments.
Calls the specified function on an object, optionally passing arguments. If the targetObject
expression evaluates to null then null will be returned.
In some cases, due to erasure, the schema may expect a primitive type when in fact the method is returning java.lang.Object. In this case, we will generate code that attempts to unbox the value automatically.
An expression that will return the object to call the method on.
The name of the method to call.
The expected return type of the function.
An optional list of expressions, whos evaluation will be passed to the function.
Evaluates to true iff it's NaN.
An expression that is evaluated to true if the input is not null.
An expression that is evaluated to true if the input is null.
A mutable wrapper that makes two rows appear as a single concatenated row.
A mutable wrapper that makes two rows appear as a single concatenated row. Designed to be instantiated once per thread and reused.
A place holder for the loop variable used in MapObjects.
A place holder for the loop variable used in MapObjects. This should never be constructed manually, but will instead be passed into the provided lambda function.
Returns the last day of the month which the date belongs to.
A leaf expression, i.e.
A leaf expression, i.e. one without any child expressions.
A leaf expression specifically for math constants.
A leaf expression specifically for math constants. Math constants expect no input.
There is no code generation because they should get constant folded by the optimizer.
A function that returns the least value of all parameters, skipping null values.
A function that returns the least value of all parameters, skipping null values. It takes at least 2 parameters, and returns null iff all parameters are null.
A function that return the length of the given string or binary expression.
A function that return the Levenshtein distance between the two given strings.
Simple RegEx pattern matching function
In order to do type checking, use Literal.create() instead of constructor
Computes the logarithm of a number.
Computes the logarithm of a number.
the logarithm base, default to e.
the number to compute the logarithm of.
A function that converts the characters of a string to lowercase.
A function that converts the characters of a string to lowercase.
Create a Decimal from an unscaled Long value.
Create a Decimal from an unscaled Long value. Note: this expression is internal and created only by the optimizer, we don't need to do type check for it.
Applies the given expression to every element of a collection of items, returning the result as an ArrayType.
Applies the given expression to every element of a collection of items, returning the result as an ArrayType. This is similar to a typical map operation, but where the lambda function is expressed using catalyst expressions.
The following collection ObjectTypes are currently supported: Seq, Array, ArrayData, java.util.List
A function that calculates an MD5 128-bit checksum and returns it as a hex string For input of type BinaryType
Returns number of months between dates date1 and date2.
Converts a InternalRow to another Row given a sequence of expression that define each column of the new row.
Converts a InternalRow to another Row given a sequence of expression that define each column of the new row. If the schema of the input row is specified, then the given expression will be bound to that schema.
In contrast to a normal projection, a MutableProjection reuses the same underlying row object
each time an input row is added. This significantly reduces the cost of calculating the
projection, but means that it is not safe to hold on to a reference to a InternalRow after
next() has been called on the Iterator that produced it. Instead, the user must call
InternalRow.copy() and hold on to the returned InternalRow before calling next().
An extended interface to InternalRow that allows the values for each column to be updated.
An extended interface to InternalRow that allows the values for each column to be updated. Setting a value through a primitive function implicitly marks that column as not null.
A parent class for mutable container objects that are reused when the values are changed, resulting in less garbage.
A parent class for mutable container objects that are reused when the values are changed, resulting in less garbage. These values are held by a SpecificMutableRow.
The following code was roughly used to generate these objects:
val types = "Int,Float,Boolean,Double,Short,Long,Byte,Any".split(",") types.map {tpe => s""" final class Mutable$tpe extends MutableValue { var value: $tpe = 0 def boxed = if (isNull) null else value def update(v: Any) = value = { isNull = false v.asInstanceOf[$tpe] } def copy() = { val newCopy = new Mutable$tpe newCopy.isNull = isNull newCopy.value = value newCopy } }""" }.foreach(println) types.map { tpe => s""" override def set$tpe(ordinal: Int, value: $tpe): Unit = { val currentValue = values(ordinal).asInstanceOf[Mutable$tpe] currentValue.isNull = false currentValue.value = value } override def get$tpe(i: Int): $tpe = { values(i).asInstanceOf[Mutable$tpe].value }""" }.foreach(println)
An Expression evaluates to left iff it's not NaN, or evaluates to right otherwise.
An Expression evaluates to left iff it's not NaN, or evaluates to right otherwise.
This Expression is useful for mapping NaN values to null.
An Expression that is named.
Constructs a new instance of the given class, using the result of evaluating the specified expressions as arguments.
Constructs a new instance of the given class, using the result of evaluating the specified expressions as arguments.
The class to construct.
A list of expression to use as arguments to the constructor.
When true, if any of the arguments is null, then null will be returned instead of trying to construct the object.
The type of object being constructed, as a Spark SQL datatype. This allows you to manually specify the type when the object in question is a valid internal representation (i.e. ArrayData) instead of an object.
If the object being constructed is an inner class the outerPointer must for the containing class must be specified.
Returns the first date which is later than startDate and named as dayOfWeek.
Returns the first date which is later than startDate and named as dayOfWeek. For example, NextDay(2015-07-27, Sunday) would return 2015-08-02, which is the first Sunday later than 2015-07-27.
Allowed "dayOfWeek" is defined in DateTimeUtils.getDayOfWeekFromString.
An expression that is nondeterministic.
Pi.
Pi. Note that there is no code generation because this is only evaluated by the optimizer during constant folding.
An Expression that returns a boolean value.
A place holder used when printing expressions without debugging information such as the expression id or the unresolved indicator.
Converts a InternalRow to another Row given a sequence of expression that define each column of the new row.
Converts a InternalRow to another Row given a sequence of expression that define each column of the new row. If the schema of the input row is specified, then the given expression will be bound to that schema.
An expression used to wrap the children when promote the precision of DecimalType to avoid promote multiple times.
A Random distribution generating expression.
A Random distribution generating expression. TODO: This can be made generic to generate any type of random distribution, or any type of StructType.
Since this expression is stateful, it cannot be a case object.
Generate a random column with i.i.d.
Generate a random column with i.i.d. uniformly distributed values in [0, 1).
Generate a random column with i.i.d.
Generate a random column with i.i.d. gaussian random distribution.
Extract a specific(idx) group identified by a Java regex.
Extract a specific(idx) group identified by a Java regex.
NOTE: this expression is not THREAD-SAFE, as it has some internal mutable status.
Replace all substrings of str that match regexp with rep.
Replace all substrings of str that match regexp with rep.
NOTE: this expression is not THREAD-SAFE, as it has some internal mutable status.
Round the child's result to scale decimal place when scale >= 0
or round at integral part when scale < 0.
Round the child's result to scale decimal place when scale >= 0
or round at integral part when scale < 0.
For example, round(31.415, 2) = 31.42 and round(31.415, -1) = 30.
Child of IntegralType would round to itself when scale >= 0.
Child of FractionalType whose value is NaN or Infinite would always round to itself.
Round's dataType would always equal to child's dataType except for DecimalType,
which would lead scale decrease from the origin DecimalType.
expr to be round, all NumericType is allowed as Input
new scale to be round to, this should be a constant int at runtime
User-defined function.
User-defined function.
The user defined scala function to run. Note that if you use primitive parameters, you are not able to check if it is null or not, and the UDF will return null for you if the primitive input is null. Use boxed type or Option if you wanna do the null-handling yourself.
Return type of function.
The input expressions of this UDF.
The expected input types of this UDF.
A function that calculates a sha1 hash value and returns it as a hex string For input of type BinaryType or StringType
A function that calculates the SHA-2 family of functions (SHA-224, SHA-256, SHA-384, and SHA-512) and returns it as a hex string.
A function that calculates the SHA-2 family of functions (SHA-224, SHA-256, SHA-384, and SHA-512) and returns it as a hex string. The first argument is the string or binary to be hashed. The second argument indicates the desired bit length of the result, which must have a value of 224, 256, 384, 512, or 0 (which is equivalent to 256). SHA-224 is supported starting from Java 8. If asking for an unsupported SHA function, the return value is NULL. If either argument is NULL or the hash length is not one of the permitted values, the return value is NULL.
Bitwise unsigned left shift.
Bitwise unsigned left shift.
the base number to shift.
number of bits to left shift.
Bitwise unsigned left shift.
Bitwise unsigned left shift.
the base number to shift.
number of bits to left shift.
Bitwise unsigned right shift, for integer and long data type.
Bitwise unsigned right shift, for integer and long data type.
the base number.
the number of bits to right shift.
Given an array or map, returns its size.
Sorts the input array in ascending / descending order according to the natural ordering of the array elements and returns it.
An expression that can be used to sort a tuple.
An expression that can be used to sort a tuple. This class extends expression primarily so that transformations over expression will descend into its child.
An expression to generate a 64-bit long prefix used in sorting.
A function that return soundex code of the given string expression.
A row type that holds an array specialized container objects, of type MutableValue, chosen based on the dataTypes of each column.
A row type that holds an array specialized container objects, of type MutableValue, chosen based on the dataTypes of each column. The intent is to decrease garbage when modifying the values of primitive columns.
A specified Window Frame.
A function that returns true if the string left starts with the string right.
Invokes a static function, returning the result.
Invokes a static function, returning the result. By default, any of the arguments being null will result in returning null instead of calling the function.
The target of the static call. This can either be the object itself (methods defined on scala objects), or the class object (static methods defined in java).
The expected return type of the function call
The name of the method to call.
An optional list of expressions to pass as arguments to the function.
When true, and any of the arguments is null, null will be returned instead of calling the function.
A function that returns the position of the first occurrence of substr in the given string.
A function that returns the position of the first occurrence of substr in the given string. Returns null if either of the arguments are null and returns 0 if substr could not be found in str.
NOTE: that this is not zero based, but 1-based index. The first character in str has index 1.
Returns str, left-padded with pad to a length of len.
A function that returns the position of the first occurrence of substr in given string after position pos.
A base trait for functions that compare two strings, returning a boolean.
Returns str, right-padded with pad to a length of len.
Returns the string which repeat the given string value n times.
Returns the reversed given string.
Returns a n spaces string.
Splits str around pat (pattern is a regular expression).
A function translate any character in the srcExpr by a character in replaceExpr.
A function translate any character in the srcExpr by a character in replaceExpr.
The characters in replaceExpr is corresponding to the characters in matchingExpr.
The translate will happen when any character in the string matching with the character
in the matchingExpr.
A function that trim the spaces from both ends for the specified string.
A function that trim the spaces from left end for given string.
A function that trim the spaces from right end for given string.
A function that takes a substring of its first argument starting at a given position.
A function that takes a substring of its first argument starting at a given position. Defined for String and Binary types.
Returns the substring from string str before count occurrences of the delimiter delim.
Returns the substring from string str before count occurrences of the delimiter delim. If count is positive, everything the left of the final delimiter (counting from left) is returned. If count is negative, every to the right of the final delimiter (counting from the right) is returned. substring_index performs a case-sensitive match when searching for delim.
An expression with three inputs and one output.
An expression with three inputs and one output. The output is by default evaluated to null if any input is evaluated to null.
Adds an interval to timestamp.
Subtracts an interval from timestamp.
Returns the date part of a timestamp or string.
Assumes given timestamp is in given timezone and converts to UTC.
Converts time string with given pattern.
Converts time string with given pattern. Deterministic version of UnixTimestamp, must have at least one parameter.
Returns date truncated to the unit specified by the format.
Converts the argument from a base 64 string to BINARY.
An expression with one input and one output.
An expression with one input and one output. The output is by default evaluated to null if the input is evaluated to null.
A unary expression specifically for math functions.
A unary expression specifically for math functions. Math Functions expect a specific type of
input format, therefore these functions extend ExpectsInputTypes.
An expression that cannot be evaluated.
An expression that cannot be evaluated. Some expressions don't live past analysis or optimization time (e.g. Star). This trait is used by those expressions.
Performs the inverse operation of HEX.
Performs the inverse operation of HEX. Resulting characters are returned as a byte array.
Converts time string with given pattern.
Converts time string with given pattern. (see [http://docs.oracle.com/javase/tutorial/i18n/format/simpleDateFormat.html]) to Unix time stamp (in seconds), returns null if fail. Note that hive Language Manual says it returns 0 if fail, but in fact it returns null. If the second parameter is missing, use "yyyy-MM-dd HH:mm:ss". If no parameters provided, the first parameter will be current_timestamp. If the first parameter is a Date or Timestamp instead of String, we will ignore the second parameter.
A projection that returns UnsafeRow.
Return the unscaled Long value of a Decimal, assuming it fits in a Long.
Return the unscaled Long value of a Decimal, assuming it fits in a Long. Note: this expression is internal and created only by the optimizer, we don't need to do type check for it.
Given an expression that returns on object of type Option[_], this expression unwraps the
option into the specified Spark SQL datatype.
Given an expression that returns on object of type Option[_], this expression unwraps the
option into the specified Spark SQL datatype. In the case of None, the nullbit is set instead.
The expected unwrapped option type.
An expression that returns an Option
Cast the child expression to the target data type, but will throw error if the cast might truncate, e.g.
Cast the child expression to the target data type, but will throw error if the cast might truncate, e.g. long -> int, timestamp -> data.
A function that converts the characters of a string to uppercase.
A function that converts the characters of a string to uppercase.
A generator that produces its output using the provided lambda function.
<value> FOLLOWING boundary.
<value> PRECEDING boundary.
The trait used to represent the a Window Frame.
Every window function needs to maintain a output buffer for its output.
Every window function needs to maintain a output buffer for its output. It should expect that for a n-row window frame, it will be called n times to retrieve value corresponding with these n rows.
The trait of the Window Specification (specified in the OVER clause or WINDOW clause) for Window Functions.
The specification for a window function.
The specification for a window function.
It defines the way that input rows are partitioned.
It defines the ordering of rows in a partition.
It defines the window frame in a partition.
A Window specification reference that refers to the WindowSpecDefinition defined
under the name name.
Converts the result of evaluating child into an option, checking both the isNull bit and
(in the case of reference types) equality with null.
Converts the result of evaluating child into an option, checking both the isNull bit and
(in the case of reference types) equality with null.
The expression to evaluate and wrap.
The type of this option.
Builds a map that is keyed by an Attribute's expression id.
Builds a map that is keyed by an Attribute's expression id. Using the expression id allows values to be looked up even when the attributes used differ cosmetically (i.e., the capitalization of the name, or the expected nullability).
CURRENT ROW boundary.
Used as input into expressions whose output does not depend on any input value.
Extractor for making working with frame boundaries easier.
A projection that could turn UnsafeRow into GenericInternalRow
Extractor for retrieving Int literals.
An extractor that matches non-null literal values
RangeFrame treats rows in a partition as groups of peers.
RangeFrame treats rows in a partition as groups of peers.
All rows having the same ORDER BY ordering are considered as peers.
When a ValuePreceding or a ValueFollowing is used as its FrameBoundary,
the value is considered as a logical offset.
For example, assuming the value of the current row's ORDER BY expression expr is v,
RANGE BETWEEN 1 PRECEDING AND 1 FOLLOWING represents a frame containing rows whose values
expr are in the range of [v-1, v+1].
If ORDER BY clause is not defined, all rows in the partition is considered as peers
of the current row.
RowFrame treats rows in a partition individually.
RowFrame treats rows in a partition individually. When a ValuePreceding
or a ValueFollowing is used as its FrameBoundary, the value is considered
as a physical offset.
For example, ROW BETWEEN 1 PRECEDING AND 1 FOLLOWING represents a 3-row frame,
from the row precedes the current row to the row follows the current row.
UNBOUNDED FOLLOWING boundary.
UNBOUNDED PRECEDING boundary.
Used as a place holder when a frame specification is not defined.
A collection of generators that build custom bytecode at runtime for performing the evaluation of catalyst expression.
A set of classes that can be used to represent trees of relational expressions. A key goal of the expression library is to hide the details of naming and scoping from developers who want to manipulate trees of relational operators. As such, the library defines a special type of expression, a NamedExpression in addition to the standard collection of expressions.
Standard Expressions
A library of standard expressions (e.g., Add, EqualTo), aggregates (e.g., SUM, COUNT), and other computations (e.g. UDFs). Each expression type is capable of determining its output schema as a function of its children's output schema.
Named Expressions
Some expression are named and thus can be referenced by later operators in the dataflow graph. The two types of named expressions are AttributeReferences and Aliases. AttributeReferences refer to attributes of the input tuple for a given operator and form the leaves of some expression trees. Aliases assign a name to intermediate computations. For example, in the SQL statement
SELECT a+b AS c FROM ..., the expressionsaandbwould be represented byAttributeReferencesandcwould be represented by anAlias.During analysis, all named expressions are assigned a globally unique expression id, which can be used for equality comparisons. While the original names are kept around for debugging purposes, they should never be used to check if two attributes refer to the same value, as plan transformations can result in the introduction of naming ambiguity. For example, consider a plan that contains subqueries, both of which are reading from the same table. If an optimization removes the subqueries, scoping information would be destroyed, eliminating the ability to reason about which subquery produced a given attribute.
Evaluation
The result of expressions can be evaluated using the
Expression.apply(Row)method.