TypedDatasetForwarded

Value Members

1. final def !=(arg0: Any): Boolean

   

   Definition Classes

   AnyRef → Any

2. final def ##(): Int

   

   Definition Classes

   AnyRef → Any

3. final def ==(arg0: Any): Boolean

   

   Definition Classes

   AnyRef → Any

4. final def asInstanceOf[T0]: T0

   

   Definition Classes

   Any

5. def cache(): TypedDataset[T]

   

   Persist this TypedDataset with the default storage level (MEMORY_AND_DISK).

   Persist this TypedDataset with the default storage level (MEMORY_AND_DISK).

   apache/spark

6. def clone(): AnyRef

   

   Attributes

   protected[java.lang]

   Definition Classes

   AnyRef

   Annotations

   @throws( ... )

7. def coalesce(numPartitions: Int): TypedDataset[T]

   

   Returns a new TypedDataset that has exactly numPartitions partitions.

   Returns a new TypedDataset that has exactly numPartitions partitions. Similar to coalesce defined on an RDD, this operation results in a narrow dependency, e.g. if you go from 1000 partitions to 100 partitions, there will not be a shuffle, instead each of the 100 new partitions will claim 10 of the current partitions.

   apache/spark

8. def distinct: TypedDataset[T]

   

   Returns a new TypedDataset that contains only the unique elements of this TypedDataset.

   Returns a new TypedDataset that contains only the unique elements of this TypedDataset.

   Note that, equality checking is performed directly on the encoded representation of the data and thus is not affected by a custom equals function defined on T.

   apache/spark

9. final def eq(arg0: AnyRef): Boolean

   

   Definition Classes

   AnyRef

10. def equals(arg0: Any): Boolean

    

    Definition Classes

    AnyRef → Any

11. def except(other: TypedDataset[T]): TypedDataset[T]

    

    Returns a new Dataset containing rows in this Dataset but not in another Dataset.

    Returns a new Dataset containing rows in this Dataset but not in another Dataset. This is equivalent to EXCEPT in SQL.

    Note that, equality checking is performed directly on the encoded representation of the data and thus is not affected by a custom equals function defined on T.

    apache/spark

12. def explain(extended: Boolean = false): Unit

    

    Prints the plans (logical and physical) to the console for debugging purposes.

    Prints the plans (logical and physical) to the console for debugging purposes.

    apache/spark

13. def filter(func: (T) ⇒ Boolean): TypedDataset[T]

    

    Returns a new TypedDataset that only contains elements where func returns true.

    Returns a new TypedDataset that only contains elements where func returns true.

    apache/spark

14. def finalize(): Unit

    

    Attributes

    protected[java.lang]

    Definition Classes

    AnyRef

    Annotations

    @throws( classOf[java.lang.Throwable] )

15. def flatMap[U](func: (T) ⇒ TraversableOnce[U])(implicit arg0: TypedEncoder[U]): TypedDataset[U]

    

    Returns a new TypedDataset by first applying a function to all elements of this TypedDataset, and then flattening the results.

    Returns a new TypedDataset by first applying a function to all elements of this TypedDataset, and then flattening the results.

    apache/spark

16. final def getClass(): Class[_]

    

    Definition Classes

    AnyRef → Any

17. def hashCode(): Int

    

    Definition Classes

    AnyRef → Any

18. def intersect(other: TypedDataset[T]): TypedDataset[T]

    

    Returns a new TypedDataset that contains only the elements of this TypedDataset that are also present in other.

    Returns a new TypedDataset that contains only the elements of this TypedDataset that are also present in other.

    Note that, equality checking is performed directly on the encoded representation of the data and thus is not affected by a custom equals function defined on T.

    apache/spark

19. final def isInstanceOf[T0]: Boolean

    

    Definition Classes

    Any

20. def limit(n: Int): TypedDataset[T]

    

    Returns a new Dataset by taking the first n rows.

    Returns a new Dataset by taking the first n rows. The difference between this function and head is that head is an action and returns an array (by triggering query execution) while limit returns a new Dataset.

    apache/spark

21. def map[U](func: (T) ⇒ U)(implicit arg0: TypedEncoder[U]): TypedDataset[U]

    

    Returns a new TypedDataset that contains the result of applying func to each element.

    Returns a new TypedDataset that contains the result of applying func to each element.

    apache/spark

22. def mapPartitions[U](func: (Iterator[T]) ⇒ Iterator[U])(implicit arg0: TypedEncoder[U]): TypedDataset[U]

    

    Returns a new TypedDataset that contains the result of applying func to each partition.

    Returns a new TypedDataset that contains the result of applying func to each partition.

    apache/spark

23. final def ne(arg0: AnyRef): Boolean

    

    Definition Classes

    AnyRef

24. final def notify(): Unit

    

    Definition Classes

    AnyRef

25. final def notifyAll(): Unit

    

    Definition Classes

    AnyRef

26. def persist(newLevel: StorageLevel = StorageLevel.MEMORY_AND_DISK): TypedDataset[T]

    

    Persist this TypedDataset with the given storage level.

    Persist this TypedDataset with the given storage level.

    newLevel

    One of: MEMORY_ONLY, MEMORY_AND_DISK, MEMORY_ONLY_SER, MEMORY_AND_DISK_SER, DISK_ONLY, MEMORY_ONLY_2, MEMORY_AND_DISK_2, etc. apache/spark

27. def printSchema(): Unit

    

    Prints the schema of the underlying Dataset to the console in a nice tree format.

    Prints the schema of the underlying Dataset to the console in a nice tree format.

    apache/spark

28. def rdd: RDD[T]

    

    Converts this TypedDataset to an RDD.

    Converts this TypedDataset to an RDD.

    apache/spark

29. def repartition(numPartitions: Int): TypedDataset[T]

    

    Returns a new TypedDataset that has exactly numPartitions partitions.

    Returns a new TypedDataset that has exactly numPartitions partitions.

    apache/spark

30. def sample(withReplacement: Boolean, fraction: Double, seed: Long = Random.nextLong): TypedDataset[T]

    

    Returns a new TypedDataset by sampling a fraction of records.

    Returns a new TypedDataset by sampling a fraction of records.

    apache/spark

31. def schema: StructType

    

    Returns the schema of this Dataset.

    Returns the schema of this Dataset.

    apache/spark

32. final def synchronized[T0](arg0: ⇒ T0): T0

    

    Definition Classes

    AnyRef

33. def toDF(): DataFrame

    

    Converts this strongly typed collection of data to generic Dataframe.

    Converts this strongly typed collection of data to generic Dataframe. In contrast to the strongly typed objects that Dataset operations work on, a Dataframe returns generic Row objects that allow fields to be accessed by ordinal or name.

    apache/spark

34. def toString(): String

    

    Definition Classes

    TypedDatasetForwarded → AnyRef → Any

35. def transform[U](t: (TypedDataset[T]) ⇒ TypedDataset[U]): TypedDataset[U]

    

    Concise syntax for chaining custom transformations.

    Concise syntax for chaining custom transformations.

    apache/spark

36. def union(other: TypedDataset[T]): TypedDataset[T]

    

    Returns a new TypedDataset that contains the elements of both this and the other TypedDataset combined.

    Returns a new TypedDataset that contains the elements of both this and the other TypedDataset combined.

    Note that, this function is not a typical set union operation, in that it does not eliminate duplicate items. As such, it is analogous to UNION ALL in SQL.

    apache/spark

37. def unpersist(blocking: Boolean = false): TypedDataset[T]

    

    Mark the TypedDataset as non-persistent, and remove all blocks for it from memory and disk.

    Mark the TypedDataset as non-persistent, and remove all blocks for it from memory and disk.

    blocking

    Whether to block until all blocks are deleted. apache/spark

38. final def wait(): Unit

    

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )

39. final def wait(arg0: Long, arg1: Int): Unit

    

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )

40. final def wait(arg0: Long): Unit

    

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )