NonAggregateFunctions

Value Members

1. final def !=(arg0: Any): Boolean

   

   Definition Classes

   AnyRef → Any

2. final def ##(): Int

   

   Definition Classes

   AnyRef → Any

3. final def ==(arg0: Any): Boolean

   

   Definition Classes

   AnyRef → Any

4. def abs[A, B, T](column: AbstractTypedColumn[T, A])(implicit i0: CatalystAbsolute[A, B], i1: TypedEncoder[B]): ThisType[T, B]

   

   Non-Aggregate function: returns the absolute value of a numeric column

   Non-Aggregate function: returns the absolute value of a numeric column

   apache/spark

5. def acos[A, T](column: AbstractTypedColumn[T, A])(implicit i0: CatalystCast[A, Double]): ThisType[T, Double]

   

   Non-Aggregate function: returns the acos of a numeric column

   Non-Aggregate function: returns the acos of a numeric column

   Spark will expect a Double value for this expression. See: https://github.com/apache/spark/blob/4a3c09601ba69f7d49d1946bb6f20f5cfe453031/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/mathExpressions.scala#L67 apache/spark

6. def arrayContains[C[_], A, T](column: AbstractTypedColumn[T, C[A]], value: A)(implicit arg0: CatalystCollection[C]): ThisType[T, Boolean]

   

   Non-Aggregate function: returns true if value is contained with in the array in the specified column

   Non-Aggregate function: returns true if value is contained with in the array in the specified column

   apache/spark

7. final def asInstanceOf[T0]: T0

   

   Definition Classes

   Any

8. def ascii[T](column: AbstractTypedColumn[T, String]): ThisType[T, Int]

   

   Non-Aggregate function: takes the first letter of a string column and returns the ascii int value in a new column

   Non-Aggregate function: takes the first letter of a string column and returns the ascii int value in a new column

   apache/spark

9. def asin[A, T](column: AbstractTypedColumn[T, A])(implicit i0: CatalystCast[A, Double]): ThisType[T, Double]

   

   Non-Aggregate function: returns the asin of a numeric column

   Non-Aggregate function: returns the asin of a numeric column

   Spark will expect a Double value for this expression. See: https://github.com/apache/spark/blob/4a3c09601ba69f7d49d1946bb6f20f5cfe453031/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/mathExpressions.scala#L67 apache/spark

10. def atan[A, T](column: AbstractTypedColumn[T, A])(implicit i0: CatalystCast[A, Double]): ThisType[T, Double]

    

    Non-Aggregate function: returns the atan of a numeric column

    Non-Aggregate function: returns the atan of a numeric column

    Spark will expect a Double value for this expression. See: https://github.com/apache/spark/blob/4a3c09601ba69f7d49d1946bb6f20f5cfe453031/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/mathExpressions.scala#L67 apache/spark

11. def atan2[A, T](l: TypedAggregate[T, A], r: Double)(implicit i0: CatalystCast[A, Double]): TypedAggregate[T, Double]

    

12. def atan2[B, T](l: Double, r: TypedAggregate[T, B])(implicit i0: CatalystCast[B, Double]): TypedAggregate[T, Double]

    

13. def atan2[A, T](l: TypedColumn[T, A], r: Double)(implicit i0: CatalystCast[A, Double]): TypedColumn[T, Double]

    

14. def atan2[B, T](l: Double, r: TypedColumn[T, B])(implicit i0: CatalystCast[B, Double]): TypedColumn[T, Double]

    

15. def atan2[A, B, T](l: TypedAggregate[T, A], r: TypedAggregate[T, B])(implicit i0: CatalystCast[A, Double], i1: CatalystCast[B, Double]): TypedAggregate[T, Double]

    

    Non-Aggregate function: returns the angle theta from the conversion of rectangular coordinates (x, y) to polar coordinates (r, theta).

    Non-Aggregate function: returns the angle theta from the conversion of rectangular coordinates (x, y) to polar coordinates (r, theta).

    Spark will expect a Double value for this expression. See: https://github.com/apache/spark/blob/4a3c09601ba69f7d49d1946bb6f20f5cfe453031/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/mathExpressions.scala#L67 apache/spark

16. def atan2[A, B, T](l: TypedColumn[T, A], r: TypedColumn[T, B])(implicit i0: CatalystCast[A, Double], i1: CatalystCast[B, Double]): TypedColumn[T, Double]

    

    Non-Aggregate function: returns the angle theta from the conversion of rectangular coordinates (x, y) to polar coordinates (r, theta).

    Non-Aggregate function: returns the angle theta from the conversion of rectangular coordinates (x, y) to polar coordinates (r, theta).

    Spark will expect a Double value for this expression. See: https://github.com/apache/spark/blob/4a3c09601ba69f7d49d1946bb6f20f5cfe453031/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/mathExpressions.scala#L67 apache/spark

17. def base64[T](column: AbstractTypedColumn[T, Array[Byte]]): ThisType[T, String]

    

    Non-Aggregate function: Computes the BASE64 encoding of a binary column and returns it as a string column.

    Non-Aggregate function: Computes the BASE64 encoding of a binary column and returns it as a string column. This is the reverse of unbase64.

    apache/spark

18. def bin[T](column: AbstractTypedColumn[T, Long]): ThisType[T, String]

    

    Non-Aggregate function: Returns the string representation of the binary value of the given long column.

    Non-Aggregate function: Returns the string representation of the binary value of the given long column. For example, bin("12") returns "1100".

    apache/spark

19. def bitwiseNOT[A, T](column: AbstractTypedColumn[T, A])(implicit arg0: CatalystBitwise[A]): ThisType[T, A]

    

    Non-Aggregate function: Computes bitwise NOT.

    Non-Aggregate function: Computes bitwise NOT.

    apache/spark

20. def clone(): AnyRef

    

    Attributes

    protected[java.lang]

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )

21. def concat[T](columns: TypedAggregate[T, String]*): TypedAggregate[T, String]

    

    Non-Aggregate function: Concatenates multiple input string columns together into a single string column.

    Non-Aggregate function: Concatenates multiple input string columns together into a single string column.

    Note

    varargs make it harder to generalize so we overload the method for TypedColumn and TypedAggregate apache/spark

22. def concat[T](columns: TypedColumn[T, String]*): TypedColumn[T, String]

    

    Non-Aggregate function: Concatenates multiple input string columns together into a single string column.

    Non-Aggregate function: Concatenates multiple input string columns together into a single string column.

    Note

    varargs make it harder to generalize so we overload the method for TypedColumn and TypedAggregate apache/spark

23. def concatWs[T](sep: String, columns: TypedColumn[T, String]*): TypedColumn[T, String]

    

    Non-Aggregate function: Concatenates multiple input string columns together into a single string column, using the given separator.

    Non-Aggregate function: Concatenates multiple input string columns together into a single string column, using the given separator.

    Note

    varargs make it harder to generalize so we overload the method for TypedColumn and TypedAggregate apache/spark

24. def concatWs[T](sep: String, columns: TypedAggregate[T, String]*): TypedAggregate[T, String]

    

    Non-Aggregate function: Concatenates multiple input string columns together into a single string column, using the given separator.

    Non-Aggregate function: Concatenates multiple input string columns together into a single string column, using the given separator.

    Note

    varargs make it harder to generalize so we overload the method for TypedColumn and TypedAggregate apache/spark

25. def cos[A, T](column: AbstractTypedColumn[T, A])(implicit i0: CatalystCast[A, Double]): ThisType[T, Double]

    

    Non-Aggregate function: Computes the cosine of the given value.

    Non-Aggregate function: Computes the cosine of the given value.

    Spark will expect a Double value for this expression. See: https://github.com/apache/spark/blob/4a3c09601ba69f7d49d1946bb6f20f5cfe453031/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/mathExpressions.scala#L67 apache/spark

26. def cosh[A, T](column: AbstractTypedColumn[T, A])(implicit i0: CatalystCast[A, Double]): ThisType[T, Double]

    

    Non-Aggregate function: Computes the hyperbolic cosine of the given value.

    Non-Aggregate function: Computes the hyperbolic cosine of the given value.

    Spark will expect a Double value for this expression. See: https://github.com/apache/spark/blob/4a3c09601ba69f7d49d1946bb6f20f5cfe453031/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/mathExpressions.scala#L67 apache/spark

27. final def eq(arg0: AnyRef): Boolean

    

    Definition Classes

    AnyRef

28. def equals(arg0: Any): Boolean

    

    Definition Classes

    AnyRef → Any

29. def finalize(): Unit

    

    Attributes

    protected[java.lang]

    Definition Classes

    AnyRef

    Annotations

    @throws( classOf[java.lang.Throwable] )

30. final def getClass(): Class[_]

    

    Definition Classes

    AnyRef → Any

31. def hashCode(): Int

    

    Definition Classes

    AnyRef → Any

32. def inputFileName[T](): TypedColumn[T, String]

    

    Non-Aggregate function: file name of the current Spark task.

    Non-Aggregate function: file name of the current Spark task. Empty string if row did not originate from a file

    apache/spark

33. def instr[T](str: AbstractTypedColumn[T, String], substring: String): ThisType[T, Int]

    

    Non-Aggregate function: Locates the position of the first occurrence of substring column in given string

    Non-Aggregate function: Locates the position of the first occurrence of substring column in given string

    Note

    The position is not zero based, but 1 based index. Returns 0 if substr could not be found in str. apache/spark

34. final def isInstanceOf[T0]: Boolean

    

    Definition Classes

    Any

35. def length[T](str: AbstractTypedColumn[T, String]): ThisType[T, Int]

    

    Non-Aggregate function: Computes the length of a given string.

    Non-Aggregate function: Computes the length of a given string.

    apache/spark

36. def levenshtein[T](l: TypedAggregate[T, String], r: TypedAggregate[T, String]): TypedAggregate[T, Int]

    

    Non-Aggregate function: Computes the Levenshtein distance of the two given string columns.

    Non-Aggregate function: Computes the Levenshtein distance of the two given string columns.

    apache/spark

37. def levenshtein[T](l: TypedColumn[T, String], r: TypedColumn[T, String]): TypedColumn[T, Int]

    

    Non-Aggregate function: Computes the Levenshtein distance of the two given string columns.

    Non-Aggregate function: Computes the Levenshtein distance of the two given string columns.

    apache/spark

38. def lower[T](str: AbstractTypedColumn[T, String]): ThisType[T, String]

    

    Non-Aggregate function: Converts a string column to lower case.

    Non-Aggregate function: Converts a string column to lower case.

    apache/spark

39. def lpad[T](str: AbstractTypedColumn[T, String], len: Int, pad: String): ThisType[T, String]

    

    Non-Aggregate function: Left-pad the string column with pad to a length of len.

    Non-Aggregate function: Left-pad the string column with pad to a length of len. If the string column is longer than len, the return value is shortened to len characters.

    apache/spark

40. def ltrim[T](str: AbstractTypedColumn[T, String]): ThisType[T, String]

    

    Non-Aggregate function: Trim the spaces from left end for the specified string value.

    Non-Aggregate function: Trim the spaces from left end for the specified string value.

    apache/spark

41. def monotonicallyIncreasingId[T](): TypedColumn[T, Long]

    

    Non-Aggregate function: generates monotonically increasing id

    Non-Aggregate function: generates monotonically increasing id

    apache/spark

42. final def ne(arg0: AnyRef): Boolean

    

    Definition Classes

    AnyRef

43. final def notify(): Unit

    

    Definition Classes

    AnyRef

44. final def notifyAll(): Unit

    

    Definition Classes

    AnyRef

45. def regexpReplace[T](str: AbstractTypedColumn[T, String], pattern: Regex, replacement: String): ThisType[T, String]

    

    Non-Aggregate function: Replace all substrings of the specified string value that match regexp with rep.

    Non-Aggregate function: Replace all substrings of the specified string value that match regexp with rep.

    apache/spark

46. def reverse[T](str: AbstractTypedColumn[T, String]): ThisType[T, String]

    

    Non-Aggregate function: Reverses the string column and returns it as a new string column.

    Non-Aggregate function: Reverses the string column and returns it as a new string column.

    apache/spark

47. def rpad[T](str: AbstractTypedColumn[T, String], len: Int, pad: String): ThisType[T, String]

    

    Non-Aggregate function: Right-pad the string column with pad to a length of len.

    Non-Aggregate function: Right-pad the string column with pad to a length of len. If the string column is longer than len, the return value is shortened to len characters.

    apache/spark

48. def rtrim[T](str: AbstractTypedColumn[T, String]): ThisType[T, String]

    

    Non-Aggregate function: Trim the spaces from right end for the specified string value.

    Non-Aggregate function: Trim the spaces from right end for the specified string value.

    apache/spark

49. def sin[A, T](column: AbstractTypedColumn[T, A])(implicit i0: CatalystCast[A, Double]): ThisType[T, Double]

    

    Non-Aggregate function: Computes the sine of the given value.

    Non-Aggregate function: Computes the sine of the given value.

    Spark will expect a Double value for this expression. See: https://github.com/apache/spark/blob/4a3c09601ba69f7d49d1946bb6f20f5cfe453031/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/mathExpressions.scala#L67 apache/spark

50. def sinh[A, T](column: AbstractTypedColumn[T, A])(implicit i0: CatalystCast[A, Double]): ThisType[T, Double]

    

    Non-Aggregate function: Computes the hyperbolic sine of the given value.

    Non-Aggregate function: Computes the hyperbolic sine of the given value.

    Spark will expect a Double value for this expression. See: https://github.com/apache/spark/blob/4a3c09601ba69f7d49d1946bb6f20f5cfe453031/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/mathExpressions.scala#L67 apache/spark

51. def substring[T](str: AbstractTypedColumn[T, String], pos: Int, len: Int): ThisType[T, String]

    

    Non-Aggregate function: Substring starts at pos and is of length len

    Non-Aggregate function: Substring starts at pos and is of length len

    apache/spark

52. final def synchronized[T0](arg0: ⇒ T0): T0

    

    Definition Classes

    AnyRef

53. def tan[A, T](column: AbstractTypedColumn[T, A])(implicit i0: CatalystCast[A, Double]): ThisType[T, Double]

    

    Non-Aggregate function: Computes the tangent of the given column.

    Non-Aggregate function: Computes the tangent of the given column.

    Spark will expect a Double value for this expression. See: https://github.com/apache/spark/blob/4a3c09601ba69f7d49d1946bb6f20f5cfe453031/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/mathExpressions.scala#L67 apache/spark

54. def tanh[A, T](column: AbstractTypedColumn[T, A])(implicit i0: CatalystCast[A, Double]): ThisType[T, Double]

    

    Non-Aggregate function: Computes the hyperbolic tangent of the given value.

    Non-Aggregate function: Computes the hyperbolic tangent of the given value.

    Spark will expect a Double value for this expression. See: https://github.com/apache/spark/blob/4a3c09601ba69f7d49d1946bb6f20f5cfe453031/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/mathExpressions.scala#L67 apache/spark

55. def toString(): String

    

    Definition Classes

    AnyRef → Any

56. def trim[T](str: AbstractTypedColumn[T, String]): ThisType[T, String]

    

    Non-Aggregate function: Trim the spaces from both ends for the specified string column.

    Non-Aggregate function: Trim the spaces from both ends for the specified string column.

    apache/spark

57. def upper[T](str: AbstractTypedColumn[T, String]): ThisType[T, String]

    

    Non-Aggregate function: Converts a string column to upper case.

    Non-Aggregate function: Converts a string column to upper case.

    apache/spark

58. final def wait(): Unit

    

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )

59. final def wait(arg0: Long, arg1: Int): Unit

    

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )

60. final def wait(arg0: Long): Unit

    

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )

61. def when[T, A](condition: AbstractTypedColumn[T, Boolean], value: AbstractTypedColumn[T, A]): When[T, A]

    

    Non-Aggregate function: Evaluates a list of conditions and returns one of multiple possible result expressions.

    Non-Aggregate function: Evaluates a list of conditions and returns one of multiple possible result expressions. If none match, otherwise is returned

    when(ds('boolField), ds('a))
      .when(ds('otherBoolField), lit(123))
      .otherwise(ds('b))

    apache/spark

62. def year[T](str: AbstractTypedColumn[T, String]): ThisType[T, Option[Int]]

    

    Non-Aggregate function: Extracts the year as an integer from a given date/timestamp/string.

    Non-Aggregate function: Extracts the year as an integer from a given date/timestamp/string.

    Differs from Column#year by wrapping it's result into an Option.

    apache/spark