Bitwise AND this expression and another expression.
Bitwise AND this expression and another expression.
df.select(df.col('colA) & (df.col('colB)))
a constant of the same type apache/spark
Bitwise AND this expression and another expression (of same type).
Bitwise AND this expression and another expression (of same type).
df.select(df.col('colA).cast[Int] & -1)
a constant of the same type apache/spark
Boolean AND.
Boolean AND.
df.filter ( df.col('a) === 1 && df.col('b) > 5)
Multiplication of this expression a constant.
Multiplication of this expression a constant.
// The following multiplies a person's height by their weight.
people.select( people.col('height) * people.col('weight) )
apache/spark
Multiplication of this expression and another expression.
Multiplication of this expression and another expression.
// The following multiplies a person's height by their weight.
people.select( people.col('height) * people.col('weight) )
apache/spark
Sum of this expression (column) with a constant.
Sum of this expression (column) with a constant.
// The following selects the sum of a person's height and weight. people.select( people('height) + 2 )
a constant of the same type apache/spark
Sum of this expression and another expression.
Sum of this expression and another expression.
// The following selects the sum of a person's height and weight.
people.select( people.col('height) + people.col('weight) )
apache/spark
Subtraction.
Subtraction. Subtract the other expression from this expression.
// The following selects the difference between people's height and their weight. people.select( people('height) - 1 )
a constant of the same type apache/spark
Subtraction.
Subtraction. Subtract the other expression from this expression.
// The following selects the difference between people's height and their weight.
people.select( people.col('height) - people.col('weight) )
apache/spark
Division this expression by another expression.
Division this expression by another expression.
// The following divides a person's height by their weight. people.select( people('height) / 2 )
a constant of the same type apache/spark
Division this expression by another expression.
Division this expression by another expression.
// The following divides a person's height by their weight.
people.select( people('height) / people('weight) )
another column of the same type apache/spark
Less than.
Less than.
// The following selects people younger than 21. df.select( df('age) < 21 )
a constant of the same type apache/spark
Less than.
Less than.
// The following selects people younger than the maxAge column.
df.select( df('age) < df('maxAge) )
another column of the same type apache/spark
Less than or equal to.
Less than or equal to.
// The following selects people younger than 22. df.select( df('age) <= 2 )
a constant of the same type apache/spark
Less than or equal to.
Less than or equal to.
// The following selects people younger or equal than the maxAge column. df.select( df('age) <= df('maxAge)
another column of the same type apache/spark
Inequality test.
Inequality test.
df.filter( df.col('a) =!= "a" )
apache/spark
Inequality test.
Inequality test.
df.filter( df.col('a) =!= df.col('b) )
apache/spark
Equality test.
Equality test.
df.filter( df.col('a) === df.col('b) )
apache/spark
Equality test.
Equality test.
df.filter( df.col('a) === 1 )
apache/spark
Greater than.
Greater than.
// The following selects people older than 21. df.select( df('age) > 21 )
another column of the same type apache/spark
Greater than.
Greater than.
// The following selects people older than the maxAge column.
df.select( df('age) > df('maxAge) )
another column of the same type apache/spark
Greater than or equal.
Greater than or equal.
// The following selects people older than 20. df.select( df('age) >= 21 )
another column of the same type apache/spark
Greater than or equal.
Greater than or equal.
// The following selects people older or equal than the maxAge column.
df.select( df('age) >= df('maxAge) )
another column of the same type apache/spark
Bitwise XOR this expression and another expression.
Bitwise XOR this expression and another expression.
df.select(df.col('colA) ^ (df.col('colB)))
a constant of the same type apache/spark
Bitwise XOR this expression and another expression (of same type).
Bitwise XOR this expression and another expression (of same type).
df.select(df.col('colA).cast[Long] ^ 1L)
a constant of the same type apache/spark
Boolean AND.
Boolean AND.
df.filter ( (df.col('a) === 1).and(df.col('b) > 5) )
Returns an ascending ordering used in sorting
Returns an ascending ordering used in sorting
apache/spark
True if the current column is between the lower bound and upper bound, inclusive.
True if the current column is between the lower bound and upper bound, inclusive.
another column of the same type
another column of the same type apache/spark
True if the current column is between the lower bound and upper bound, inclusive.
True if the current column is between the lower bound and upper bound, inclusive.
a constant of the same type
a constant of the same type apache/spark
Bitwise AND this expression and another expression.
Bitwise AND this expression and another expression.
df.select(df.col('colA) bitwiseAND (df.col('colB)))
Bitwise AND this expression and another expression.
Bitwise AND this expression and another expression.
df.select(df.col('colA) bitwiseAND (df.col('colB)))
a constant of the same type apache/spark
Bitwise OR this expression and another expression.
Bitwise OR this expression and another expression.
df.select(df.col('colA) bitwiseOR (df.col('colB)))
a constant of the same type apache/spark
Bitwise OR this expression and another expression.
Bitwise OR this expression and another expression.
df.select(df.col('colA) bitwiseOR (df.col('colB)))
a constant of the same type apache/spark
Bitwise XOR this expression and another expression.
Bitwise XOR this expression and another expression.
df.select(df.col('colA) bitwiseXOR (df.col('colB)))
a constant of the same type apache/spark
Bitwise XOR this expression and another expression.
Bitwise XOR this expression and another expression.
df.select(df.col('colA) bitwiseXOR (df.col('colB)))
a constant of the same type apache/spark
Casts the column to a different type.
Casts the column to a different type.
df.select(df('a).cast[Int])
String contains.
String contains.
df.filter ( df.col('a).contains(df.col('b) )
a column which values is used as a string that is being tested against. apache/spark
String contains another string literal.
String contains another string literal.
df.filter ( df.col('a).contains("foo") )
a string that is being tested against. apache/spark
Returns a descending ordering used in sorting
Returns a descending ordering used in sorting
apache/spark
Division this expression by another expression.
Division this expression by another expression.
// The following divides a person's height by their weight.
people.select( people('height) / people('weight) )
another column of the same type apache/spark
String ends with.
String ends with.
df.filter ( df.col('a).endsWith(df.col('b))
a column which values is used as a suffix that is being tested against. apache/spark
String ends with another string literal.
String ends with another string literal.
df.filter ( df.col('a).endsWith("foo")
a suffix that is being tested against. apache/spark
Convert an Optional column by providing a default value
Convert an Optional column by providing a default value
df( df('opt).getOrElse(defaultConstant) )
Convert an Optional column by providing a default value
Convert an Optional column by providing a default value
df( df('opt).getOrElse(df('defaultValue)) )
True if the current expression is an Option and it's None.
True if the current expression is an Option and it's None.
apache/spark
True if the current expression is an Option and it's not None.
True if the current expression is an Option and it's not None.
apache/spark
Returns true if the value of this column is contained in of the arguments.
Returns true if the value of this column is contained in of the arguments.
// The following selects people with age 15, 20, or 30. df.select( df('age).isin(15, 20, 30) )
are constants of the same type apache/spark
Creates a typed column of either TypedColumn or TypedAggregate.
Creates a typed column of either TypedColumn or TypedAggregate.
Subtraction.
Subtraction. Subtract the other expression from this expression.
// The following selects the difference between people's height and their weight.
people.select( people.col('height) minus people.col('weight) )
apache/spark
Multiplication of this expression and another expression.
Multiplication of this expression and another expression.
// The following multiplies a person's height by their weight.
people.select( people.col('height) multiply people.col('weight) )
apache/spark
Boolean OR.
Boolean OR.
df.filter ( (df.col('a) === 1).or(df.col('b) > 5) )
Sum of this expression and another expression.
Sum of this expression and another expression.
// The following selects the sum of a person's height and weight.
people.select( people.col('height) plus people.col('weight) )
apache/spark
String starts with.
String starts with.
df.filter ( df.col('a).startsWith(df.col('b))
a column which values is used as a prefix that is being tested against. apache/spark
String starts with another string literal.
String starts with another string literal.
df.filter ( df.col('a).startsWith("foo")
a prefix that is being tested against. apache/spark
An expression that returns a substring
An expression that returns a substring
df.select(df('a).substr(df('b), df('c)))
expression for the starting position
expression for the length of the substring
An expression that returns a substring
An expression that returns a substring
df.select(df('a).substr(0, 5))
starting position
length of the substring
Creates a typed column of either TypedColumn or TypedAggregate.
Creates a typed column of either TypedColumn or TypedAggregate.
Creates a typed column of either TypedColumn or TypedAggregate from an expression.
Creates a typed column of either TypedColumn or TypedAggregate from an expression.
Unary minus, i.e.
Unary minus, i.e. negate the expression.
// Select the amount column and negates all values.
df.select( -df('amount) )
apache/spark
Fall back to an untyped Column
Fall back to an untyped Column
Bitwise OR this expression and another expression.
Bitwise OR this expression and another expression.
df.select(df.col('colA) | (df.col('colB)))
a constant of the same type apache/spark
Bitwise OR this expression and another expression (of same type).
Bitwise OR this expression and another expression (of same type).
df.select(df.col('colA).cast[Long] | 1L)
a constant of the same type apache/spark
Boolean OR.
Boolean OR.
df.filter ( df.col('a) === 1 || df.col('b) > 5)
Expression used in
select
-like constructions.