Packages

  • package root
    Definition Classes
    root
  • package frameless
    Definition Classes
    root
  • trait TypedDatasetForwarded[T] extends AnyRef

    This trait implements TypedDataset methods that have the same signature than their Dataset equivalent.

    This trait implements TypedDataset methods that have the same signature than their Dataset equivalent. Each method simply forwards the call to the underlying Dataset.

    Documentation marked "apache/spark" is thanks to apache/spark Contributors at https://github.com/apache/spark, licensed under Apache v2.0 available at http://www.apache.org/licenses/LICENSE-2.0

    Definition Classes
    frameless
  • deserialized

object deserialized

Methods on TypedDataset[T] that go through a full serialization and deserialization of T, and execute outside of the Catalyst runtime.

Example:

  1. The correct way to do a projection on a single column is to use the select method as follows:

    ds: TypedDataset[(String, String, String)] -> ds.select(ds('_2)).run()

    Spark provides an alternative way to obtain the same resulting Dataset, using the map method:

    ds: TypedDataset[(String, String, String)] -> ds.deserialized.map(_._2).run()

    This second approach is however substantially slower than the first one, and should be avoided as possible. Indeed, under the hood this map will deserialize the entire Tuple3 to an full JVM object, call the apply method of the _._2 closure on it, and serialize the resulting String back to its Catalyst representation.

Linear Supertypes
AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. deserialized
  2. AnyRef
  3. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##: Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.CloneNotSupportedException]) @native()
  6. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  7. def equals(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef → Any
  8. def filter(func: (T) => Boolean): TypedDataset[T]

    Returns a new TypedDataset that only contains elements where func returns true.

    Returns a new TypedDataset that only contains elements where func returns true.

    apache/spark

  9. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.Throwable])
  10. def flatMap[U](func: (T) => TraversableOnce[U])(implicit arg0: TypedEncoder[U]): TypedDataset[U]

    Returns a new TypedDataset by first applying a function to all elements of this TypedDataset, and then flattening the results.

    Returns a new TypedDataset by first applying a function to all elements of this TypedDataset, and then flattening the results.

    apache/spark

  11. final def getClass(): Class[_ <: AnyRef]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  12. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  13. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  14. def map[U](func: (T) => U)(implicit arg0: TypedEncoder[U]): TypedDataset[U]

    Returns a new TypedDataset that contains the result of applying func to each element.

    Returns a new TypedDataset that contains the result of applying func to each element.

    apache/spark

  15. def mapPartitions[U](func: (Iterator[T]) => Iterator[U])(implicit arg0: TypedEncoder[U]): TypedDataset[U]

    Returns a new TypedDataset that contains the result of applying func to each partition.

    Returns a new TypedDataset that contains the result of applying func to each partition.

    apache/spark

  16. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  17. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  18. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  19. def reduceOption[F[_]](func: (T, T) => T)(implicit F: SparkDelay[F]): F[Option[T]]

    Optionally reduces the elements of this TypedDataset using the specified binary function.

    Optionally reduces the elements of this TypedDataset using the specified binary function. The given func must be commutative and associative or the result may be non-deterministic.

    Differs from Dataset#reduce by wrapping its result into an Option and an effect-suspending F.

  20. final def synchronized[T0](arg0: => T0): T0
    Definition Classes
    AnyRef
  21. def toString(): String
    Definition Classes
    AnyRef → Any
  22. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  23. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  24. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException]) @native()

Inherited from AnyRef

Inherited from Any

Ungrouped