org.apache.spark.sql.catalyst.analysis
TypeCoercion
Related Doc:
package analysis
Changes numeric values to booleans so that expressions like true = 1 can be evaluated.
Coerces the type of different branches of a CASE WHEN statement to a common type.
Turns Add/Subtract of DateType/TimestampType/StringType and CalendarIntervalType to TimeAdd/TimeSub
Hive only performs integral division with the DIV operator.
Hive only performs integral division with the DIV operator. The arguments to / are always converted to fractional types.
This ensure that the types for various functions are as expected.
Coerces the type of different branches of If statement to a common type.
Casts types according to the expected input types for Expressions.
Handles type coercion for both IN expression with subquery and IN expressions without subquery.
Handles type coercion for both IN expression with subquery and IN expressions without subquery. 1. In the first case, find the common type by comparing the left hand side (LHS) expression types against corresponding right hand side (RHS) expression derived from the subquery expression's plan output. Inject appropriate casts in the LHS and RHS side of IN expression.
2. In the second case, convert the value and in list expressions to the common operator type by looking at all the argument types and finding the closest one that all the arguments can be cast to. When no common operator type is found the original expression will be returned and an Analysis Exception will be raised at the type checking phase.
Promotes strings that appear in arithmetic expressions.
Applies any changes to AttributeReference data types that are made by other rules to instances higher in the query tree.
Widens numeric types and converts strings to numbers when appropriate.
Widens numeric types and converts strings to numbers when appropriate.
Loosely based on rules from "Hadoop: The Definitive Guide" 2nd edition, by Tom White
The implicit conversion rules can be summarized as follows:
Additionally, all types when UNION-ed with strings will be promoted to strings. Other string conversions are handled by PromoteStrings.
Widening types might result in loss of precision in the following cases: - IntegerType to FloatType - LongType to FloatType - LongType to DoubleType - DecimalType to Double
This rule is only applied to Union/Except/Intersect
This function determines the target type of a comparison operator when one operand is a String and the other is not.
This function determines the target type of a comparison operator when one operand is a String and the other is not. It also handles when one op is a Date and the other is a Timestamp by making the target type to be String.
Case 1 type widening (see the classdoc comment above for TypeCoercion).
Case 1 type widening (see the classdoc comment above for TypeCoercion).
Find the tightest common type of two types that might be used in a binary expression. This handles all numeric types except fixed-precision decimals interacting with each other or with primitive types, because in that case the precision and scale of the result depends on the operation. Those rules are implemented in DecimalPrecision.
A collection of Rule that can be used to coerce differing types that participate in operations into compatible ones.
Notes about type widening / tightest common types: Broadly, there are two cases when we need to widen data types (e.g. union, binary comparison). In case 1, we are looking for a common data type for two or more data types, and in this case no loss of precision is allowed. Examples include type inference in JSON (e.g. what's the column's data type if one row is an integer while the other row is a long?). In case 2, we are looking for a widened data type with some acceptable loss of precision (e.g. there is no common type for double and decimal because double's range is larger than decimal, and yet decimal is more precise than double, but in union we would cast the decimal into double).