DataFrameNaFunctions

Value Members

1. final def !=(arg0: Any): Boolean

   

   Definition Classes

   AnyRef → Any

2. final def ##(): Int

   

   Definition Classes

   AnyRef → Any

3. final def ==(arg0: Any): Boolean

   

   Definition Classes

   AnyRef → Any

4. final def asInstanceOf[T0]: T0

   

   Definition Classes

   Any

5. def clone(): AnyRef

   

   Attributes

   protected[java.lang]

   Definition Classes

   AnyRef

   Annotations

   @throws( ... )

6. def drop(minNonNulls: Int, cols: Seq[String]): TryAnalysis[DataFrame]

   

   Returns a new DataFrame that drops rows containing less than minNonNulls non-null and non-NaN values in the specified columns.

   Returns a new DataFrame that drops rows containing less than minNonNulls non-null and non-NaN values in the specified columns.

   Since

   1.3.1

7. def drop(how: String, cols: Seq[String]): TryAnalysis[DataFrame]

   

   Returns a new DataFrame that drops rows containing null or NaN values in the specified columns.

   Returns a new DataFrame that drops rows containing null or NaN values in the specified columns.

   If how is "any", then drop rows containing any null or NaN values in the specified columns. If how is "all", then drop rows only if every specified column is null or NaN for that row.

   Since

   1.3.1

8. def drop(cols: Seq[String]): TryAnalysis[DataFrame]

   

   Returns a new DataFrame that drops rows containing any null or NaN values in the specified columns.

   Returns a new DataFrame that drops rows containing any null or NaN values in the specified columns.

   Since

   1.3.1

9. def drop(minNonNulls: Int): DataFrame

   

   Returns a new DataFrame that drops rows containing less than minNonNulls non-null and non-NaN values.

   Returns a new DataFrame that drops rows containing less than minNonNulls non-null and non-NaN values.

   Since

   1.3.1

10. def drop(how: String): DataFrame

    

    Returns a new DataFrame that drops rows containing null or NaN values.

    Returns a new DataFrame that drops rows containing null or NaN values.

    If how is "any", then drop rows containing any null or NaN values. If how is "all", then drop rows only if every column is null or NaN for that row.

    Since

    1.3.1

11. def drop: DataFrame

    

    Returns a new DataFrame that drops rows containing any null or NaN values.

    Returns a new DataFrame that drops rows containing any null or NaN values.

    Since

    1.3.1

12. final def eq(arg0: AnyRef): Boolean

    

    Definition Classes

    AnyRef

13. def fill(value: Boolean, cols: Seq[String]): TryAnalysis[DataFrame]

    

    Returns a new DataFrame that replaces null values in specified boolean columns.

    Returns a new DataFrame that replaces null values in specified boolean columns. If a specified column is not a boolean column, it is ignored.

    Since

    2.3.0

14. def fill(value: String, cols: Seq[String]): TryAnalysis[DataFrame]

    

    Returns a new DataFrame that replaces null values in specified string columns.

    Returns a new DataFrame that replaces null values in specified string columns. If a specified column is not a string column, it is ignored.

    Since

    1.3.1

15. def fill(value: Double, cols: Seq[String]): TryAnalysis[DataFrame]

    

    Returns a new DataFrame that replaces null or NaN values in specified numeric columns.

    Returns a new DataFrame that replaces null or NaN values in specified numeric columns. If a specified column is not a numeric column, it is ignored.

    Since

    1.3.1

16. def fill(value: Long, cols: Seq[String]): TryAnalysis[DataFrame]

    

    Returns a new DataFrame that replaces null or NaN values in specified numeric columns.

    Returns a new DataFrame that replaces null or NaN values in specified numeric columns. If a specified column is not a numeric column, it is ignored.

    Since

    2.2.0

17. def fill(valueMap: Map[String, Any]): DataFrame

    

    Returns a new DataFrame that replaces null values.

    Returns a new DataFrame that replaces null values.

    The key of the map is the column name, and the value of the map is the replacement value. The value must be of the following type: Int, Long, Float, Double, String, Boolean. Replacement values are cast to the column data type.

    For example, the following replaces null values in column "A" with string "unknown", and null values in column "B" with numeric value 1.0.

    df.na.fill(Map(
      "A" -> "unknown",
      "B" -> 1.0
    ))

    Since

    1.3.1

18. def fill(value: Boolean): DataFrame

    

    Returns a new DataFrame that replaces null values in boolean columns with value.

    Returns a new DataFrame that replaces null values in boolean columns with value.

    Since

    2.3.0

19. def fill(value: String): DataFrame

    

    Returns a new DataFrame that replaces null values in string columns with value.

    Returns a new DataFrame that replaces null values in string columns with value.

    Since

    1.3.1

20. def fill(value: Double): DataFrame

    

    Returns a new DataFrame that replaces null or NaN values in numeric columns with value.

    Returns a new DataFrame that replaces null or NaN values in numeric columns with value.

    Since

    1.3.1

21. def fill(value: Long): DataFrame

    

    Returns a new DataFrame that replaces null or NaN values in numeric columns with value.

    Returns a new DataFrame that replaces null or NaN values in numeric columns with value.

    Since

    2.2.0

22. def finalize(): Unit

    

    Attributes

    protected[java.lang]

    Definition Classes

    AnyRef

    Annotations

    @throws( classOf[java.lang.Throwable] )

23. final def getClass(): Class[_]

    

    Definition Classes

    AnyRef → Any

24. final def isInstanceOf[T0]: Boolean

    

    Definition Classes

    Any

25. final def ne(arg0: AnyRef): Boolean

    

    Definition Classes

    AnyRef

26. final def notify(): Unit

    

    Definition Classes

    AnyRef

27. final def notifyAll(): Unit

    

    Definition Classes

    AnyRef

28. def replace[T](cols: Seq[String], replacement: Map[T, T]): TryAnalysis[DataFrame]

    

    Replaces values matching keys in replacement map.

    Replaces values matching keys in replacement map.

    // Replaces all occurrences of 1.0 with 2.0 in column "height" and "weight".
    df.na.replace("height" :: "weight" :: Nil, Map(1.0 -> 2.0));
    
    // Replaces all occurrences of "UNKNOWN" with "unnamed" in column "firstname" and "lastname".
    df.na.replace("firstname" :: "lastname" :: Nil, Map("UNKNOWN" -> "unnamed"));

    cols

    list of columns to apply the value replacement. If col is "*", replacement is applied on all string, numeric or boolean columns.

    replacement

    value replacement map. Key and value of replacement map must have the same type, and can only be doubles, strings or booleans. The map value can have nulls.

    Since

    1.3.1

29. def replace[T](col: String, replacement: Map[T, T]): TryAnalysis[DataFrame]

    

    Replaces values matching keys in replacement map.

    Replaces values matching keys in replacement map.

    // Replaces all occurrences of 1.0 with 2.0 in column "height".
    df.na.replace("height", Map(1.0 -> 2.0));
    
    // Replaces all occurrences of "UNKNOWN" with "unnamed" in column "name".
    df.na.replace("name", Map("UNKNOWN" -> "unnamed"));
    
    // Replaces all occurrences of "UNKNOWN" with "unnamed" in all string columns.
    df.na.replace("*", Map("UNKNOWN" -> "unnamed"));

    col

    name of the column to apply the value replacement. If col is "*", replacement is applied on all string, numeric or boolean columns.

    replacement

    value replacement map. Key and value of replacement map must have the same type, and can only be doubles, strings or booleans. The map value can have nulls.

    Since

    1.3.1

30. final def synchronized[T0](arg0: ⇒ T0): T0

    

    Definition Classes

    AnyRef

31. val underlying: org.apache.spark.sql.DataFrameNaFunctions

    

32. def unpack[U](f: (org.apache.spark.sql.DataFrameNaFunctions) ⇒ org.apache.spark.sql.Dataset[U]): Dataset[U]

    

    Unpack the underlying DataFrameNaFunctions into a DataFrame.

33. def unpackWithAnalysis[U](f: (org.apache.spark.sql.DataFrameNaFunctions) ⇒ org.apache.spark.sql.Dataset[U]): TryAnalysis[Dataset[U]]

    

    Unpack the underlying DataFrameNaFunctions into a DataFrame, it is used for transformations that can fail due to an AnalysisException.

34. final def wait(): Unit

    

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )

35. final def wait(arg0: Long, arg1: Int): Unit

    

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )

36. final def wait(arg0: Long): Unit

    

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )