TypedDatasetForwarded

Value Members

1. final def !=(arg0: Any): Boolean

   

   Definition Classes

   AnyRef → Any

2. final def ##(): Int

   

   Definition Classes

   AnyRef → Any

3. final def ==(arg0: Any): Boolean

   

   Definition Classes

   AnyRef → Any

4. final def asInstanceOf[T0]: T0

   

   Definition Classes

   Any

5. def cache(): TypedDataset[T]

   

   Persist this TypedDataset with the default storage level (MEMORY_AND_DISK).

   Persist this TypedDataset with the default storage level (MEMORY_AND_DISK).

   apache/spark

6. def clone(): AnyRef

   

   Attributes

   protected[java.lang]

   Definition Classes

   AnyRef

   Annotations

   @throws( ... )

7. def coalesce(numPartitions: Int): TypedDataset[T]

   

   Returns a new TypedDataset that has exactly numPartitions partitions.

   Returns a new TypedDataset that has exactly numPartitions partitions. Similar to coalesce defined on an RDD, this operation results in a narrow dependency, e.g. if you go from 1000 partitions to 100 partitions, there will not be a shuffle, instead each of the 100 new partitions will claim 10 of the current partitions.

   apache/spark

8. def distinct: TypedDataset[T]

   

   Returns a new TypedDataset that contains only the unique elements of this TypedDataset.

   Returns a new TypedDataset that contains only the unique elements of this TypedDataset.

   Note that, equality checking is performed directly on the encoded representation of the data and thus is not affected by a custom equals function defined on T.

   apache/spark

9. final def eq(arg0: AnyRef): Boolean

   

   Definition Classes

   AnyRef

10. def equals(arg0: Any): Boolean

    

    Definition Classes

    AnyRef → Any

11. def explain(extended: Boolean = false): Unit

    

    Prints the plans (logical and physical) to the console for debugging purposes.

    Prints the plans (logical and physical) to the console for debugging purposes.

    apache/spark

12. def filter(func: (T) ⇒ Boolean): TypedDataset[T]

    

    Returns a new TypedDataset that only contains elements where func returns true.

    Returns a new TypedDataset that only contains elements where func returns true.

    apache/spark

13. def finalize(): Unit

    

    Attributes

    protected[java.lang]

    Definition Classes

    AnyRef

    Annotations

    @throws( classOf[java.lang.Throwable] )

14. def flatMap[U](func: (T) ⇒ TraversableOnce[U])(implicit arg0: TypedEncoder[U]): TypedDataset[U]

    

    Returns a new TypedDataset by first applying a function to all elements of this TypedDataset, and then flattening the results.

    Returns a new TypedDataset by first applying a function to all elements of this TypedDataset, and then flattening the results.

    apache/spark

15. final def getClass(): Class[_]

    

    Definition Classes

    AnyRef → Any

16. def hashCode(): Int

    

    Definition Classes

    AnyRef → Any

17. def intersect(other: TypedDataset[T]): TypedDataset[T]

    

    Returns a new TypedDataset that contains only the elements of this TypedDataset that are also present in other.

    Returns a new TypedDataset that contains only the elements of this TypedDataset that are also present in other.

    Note that, equality checking is performed directly on the encoded representation of the data and thus is not affected by a custom equals function defined on T.

    apache/spark

18. final def isInstanceOf[T0]: Boolean

    

    Definition Classes

    Any

19. def map[U](func: (T) ⇒ U)(implicit arg0: TypedEncoder[U]): TypedDataset[U]

    

    Returns a new TypedDataset that contains the result of applying func to each element.

    Returns a new TypedDataset that contains the result of applying func to each element.

    apache/spark

20. def mapPartitions[U](func: (Iterator[T]) ⇒ Iterator[U])(implicit arg0: TypedEncoder[U]): TypedDataset[U]

    

    Returns a new TypedDataset that contains the result of applying func to each partition.

    Returns a new TypedDataset that contains the result of applying func to each partition.

    apache/spark

21. final def ne(arg0: AnyRef): Boolean

    

    Definition Classes

    AnyRef

22. final def notify(): Unit

    

    Definition Classes

    AnyRef

23. final def notifyAll(): Unit

    

    Definition Classes

    AnyRef

24. def persist(newLevel: StorageLevel = StorageLevel.MEMORY_AND_DISK): TypedDataset[T]

    

    Persist this TypedDataset with the given storage level.

    Persist this TypedDataset with the given storage level.

    newLevel

    One of: MEMORY_ONLY, MEMORY_AND_DISK, MEMORY_ONLY_SER, MEMORY_AND_DISK_SER, DISK_ONLY, MEMORY_ONLY_2, MEMORY_AND_DISK_2, etc. apache/spark

25. def printSchema(): Unit

    

    Prints the schema of the underlying Dataset to the console in a nice tree format.

    Prints the schema of the underlying Dataset to the console in a nice tree format.

    apache/spark

26. def rdd: RDD[T]

    

    Converts this TypedDataset to an RDD.

    Converts this TypedDataset to an RDD.

    apache/spark

27. def repartition(numPartitions: Int): TypedDataset[T]

    

    Returns a new TypedDataset that has exactly numPartitions partitions.

    Returns a new TypedDataset that has exactly numPartitions partitions.

    apache/spark

28. def sample(withReplacement: Boolean, fraction: Double, seed: Long = Random.nextLong): TypedDataset[T]

    

    Returns a new TypedDataset by sampling a fraction of records.

    Returns a new TypedDataset by sampling a fraction of records.

    apache/spark

29. def subtract(other: TypedDataset[T]): TypedDataset[T]

    

    Returns a new TypedDataset where any elements present in other have been removed.

    Returns a new TypedDataset where any elements present in other have been removed.

    Note that, equality checking is performed directly on the encoded representation of the data and thus is not affected by a custom equals function defined on T.

    apache/spark

30. final def synchronized[T0](arg0: ⇒ T0): T0

    

    Definition Classes

    AnyRef

31. def toDF(): DataFrame

    

    Converts this strongly typed collection of data to generic Dataframe.

    Converts this strongly typed collection of data to generic Dataframe. In contrast to the strongly typed objects that Dataset operations work on, a Dataframe returns generic Row objects that allow fields to be accessed by ordinal or name.

    apache/spark

32. def toString(): String

    

    Definition Classes

    TypedDatasetForwarded → AnyRef → Any

33. def transform[U](t: (TypedDataset[T]) ⇒ TypedDataset[U]): TypedDataset[U]

    

    Concise syntax for chaining custom transformations.

    Concise syntax for chaining custom transformations.

    apache/spark

34. def union(other: TypedDataset[T]): TypedDataset[T]

    

    Returns a new TypedDataset that contains the elements of both this and the other TypedDataset combined.

    Returns a new TypedDataset that contains the elements of both this and the other TypedDataset combined.

    Note that, this function is not a typical set union operation, in that it does not eliminate duplicate items. As such, it is analogous to UNION ALL in SQL.

    apache/spark

35. def unpersist(blocking: Boolean = false): TypedDataset[T]

    

    Mark the TypedDataset as non-persistent, and remove all blocks for it from memory and disk.

    Mark the TypedDataset as non-persistent, and remove all blocks for it from memory and disk.

    blocking

    Whether to block until all blocks are deleted. apache/spark

36. final def wait(): Unit

    

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )

37. final def wait(arg0: Long, arg1: Int): Unit

    

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )

38. final def wait(arg0: Long): Unit

    

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )