Trait/Object

io.eels.datastream

DataStream

Related Docs: object DataStream | package datastream

Permalink

trait DataStream extends Logging

A DataStream is kind of like a table of data. It has fields (like columns) and rows of data. Each row has an entry for each field (this may be null depending on the field definition).

It is a lazily evaluated data structure. Each operation on a stream will create a new derived stream, but those operations will only occur when a final action is performed.

You can create a DataStream from an IO source, such as a Parquet file or a Hive table, or you may create a fully evaluated one from an in memory structure. In the case of the former, the data will only be loaded on demand as an action is performed.

A DataStream is split into one or more flows. Each flow can operate independantly of the others. For example, if you filter a flow, each flow will be filtered seperately, which allows it to be parallelized. If you write out a flow, each partition can be written out to individual files, again allowing parallelization.

Self Type
DataStream
Linear Supertypes
Logging, AnyRef, Any
Known Subclasses
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. DataStream
  2. Logging
  3. AnyRef
  4. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Abstract Value Members

  1. abstract def schema: StructType

    Permalink
  2. abstract def subscribe(subscriber: Subscriber[Seq[Row]]): Unit

    Permalink

Concrete Value Members

  1. final def !=(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int

    Permalink
    Definition Classes
    AnyRef → Any
  3. def ++(other: DataStream): DataStream

    Permalink

    Joins two streams together, such that the elements of the given datastream are appended to the end of this datastream.

  4. final def ==(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  5. def addField(field: Field, defaultValue: Any, errorIfFieldExists: Boolean): DataStream

    Permalink
  6. def addField(name: Field, defaultValue: Any): DataStream

    Permalink

    Returns a new DataStream with the given field added at the end.

    Returns a new DataStream with the given field added at the end. The value of this field for each Row is specified by the default value. The value must be compatible with the field definition. Eg, an error will occur if the field has type Int and the default value was 1.3

  7. def addField(field: Field, expression: Expression, errorIfFieldExists: Boolean): DataStream

    Permalink
  8. def addField(field: Field, expression: Expression): DataStream

    Permalink
  9. def addField(name: String, defaultValue: String, errorIfFieldExists: Boolean): DataStream

    Permalink
  10. def addField(name: String, defaultValue: String): DataStream

    Permalink

    Returns a new DataStream with the new field of type String added at the end.

    Returns a new DataStream with the new field of type String added at the end. The value of this field for each Row is specified by the default value.

  11. def addFieldFn(name: String, fn: (Row) ⇒ Any, errorIfFieldExists: Boolean): DataStream

    Permalink
  12. def addFieldFn(name: String, fn: (Row) ⇒ Any): DataStream

    Permalink
  13. def addFieldFn(field: Field, fn: (Row) ⇒ Any, errorIfFieldExists: Boolean): DataStream

    Permalink
  14. def addFieldFn(field: Field, fn: (Row) ⇒ Any): DataStream

    Permalink

    Returns a new DataStream with a new field added at the end.

    Returns a new DataStream with a new field added at the end. The value for the field is taken from the function which is invoked for each row.

  15. def aggregated(): GroupedDataStream

    Permalink
  16. def align(_schema: StructType): DataStream

    Permalink

    Accepts a schema and 'aligns' this datastream to match the schema.

    Accepts a schema and 'aligns' this datastream to match the schema. In this sense, align means the values of each row will be re-ordered to match the schema, and extraneous fields will be dropped. Any missing values will cause an exception to the thrown.

    For example, given a DataStream of schema a,b,c and align is called with a schema of c,a then the row with values (1,2,3) would become (3,1).

    If a DataStream of schema a,b,c was invoked with align with the schema d,a then an exception would be raised because d was not in the original schema.

  17. final def asInstanceOf[T0]: T0

    Permalink
    Definition Classes
    Any
  18. def cartesian(other: DataStream): DataStream

    Permalink

    Returns a new DataStream which is the result of joining every row in this datastream with every row in the given datastream.

    Returns a new DataStream which is the result of joining every row in this datastream with every row in the given datastream.

    The given datastream will be materialized before it is used.

    For example, if this datastream has rows [a,b], [c,d] and [e,f] and the given datastream has [1,2] and [3,4] then the result will be [a,b,1,2], [a,b,3,4], [c,d,1,2], [c,d,3,4], [e,f,1,2] and [e,f,3,4].

  19. def clone(): AnyRef

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  20. def collect: Vector[Row]

    Permalink

    Action which results in all the rows being returned in memory as a Vector.

  21. def collectValues: Vector[Seq[Any]]

    Permalink
  22. def concat(other: DataStream): DataStream

    Permalink

    Combines two datastreams together such that the fields from this datastream are joined with the fields of the given datastream.

    Combines two datastreams together such that the fields from this datastream are joined with the fields of the given datastream. Eg, if this datastream has fields A,B and the given datastream has fields C,D then the result will have fields A,B,C,D

    This operation requires an executor, as it must buffer rows to ensure an even distribution.

  23. def count: Long

    Permalink
  24. def drop(n: Int): DataStream

    Permalink
  25. def dropField(fieldName: String, caseSensitive: Boolean = true): DataStream

    Permalink
  26. def dropFieldIfExists(fieldName: String, caseSensitive: Boolean = true): DataStream

    Permalink
  27. def dropFields(regex: Regex): DataStream

    Permalink
  28. def dropNullRows(): DataStream

    Permalink
  29. def dropWhile(fieldName: String, p: (Any) ⇒ Boolean): DataStream

    Permalink
  30. def dropWhile(p: (Row) ⇒ Boolean): DataStream

    Permalink
  31. final def eq(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  32. def equals(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  33. def exists(p: (Row) ⇒ Boolean): Boolean

    Permalink
  34. def explode(fn: (Row) ⇒ Seq[Row]): DataStream

    Permalink
  35. def filter(expression: Equals): DataStream

    Permalink
  36. def filter(fieldName: String, p: (Any) ⇒ Boolean): DataStream

    Permalink

    Filters where the given field name matches the given predicate.

  37. def filter(f: (Row) ⇒ Boolean): DataStream

    Permalink
  38. def filterNot(p: (Row) ⇒ Boolean): DataStream

    Permalink
  39. def finalize(): Unit

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  40. def find(p: (Row) ⇒ Boolean): Option[Row]

    Permalink
  41. def foreach[U](fn: (Row) ⇒ U): DataStream

    Permalink

    Execute a side effecting function for every row in the stream, returning the same row.

  42. final def getClass(): Class[_]

    Permalink
    Definition Classes
    AnyRef → Any
  43. def groupBy(fn: (Row) ⇒ Any): GroupedDataStream

    Permalink
  44. def groupBy(fields: Iterable[String]): GroupedDataStream

    Permalink
  45. def groupBy(first: String, rest: String*): GroupedDataStream

    Permalink
  46. def hashCode(): Int

    Permalink
    Definition Classes
    AnyRef → Any
  47. def head: Row

    Permalink
  48. def intersection(stream: DataStream): DataStream

    Permalink
  49. final def isInstanceOf[T0]: Boolean

    Permalink
    Definition Classes
    Any
  50. def iterator: Iterator[Row]

    Permalink
  51. def join(key: String, other: DataStream): DataStream

    Permalink

    Joins the given datastream to this datastream on the given key column, where the values of the keys are equal as taken by the scala == operator.

    Joins the given datastream to this datastream on the given key column, where the values of the keys are equal as taken by the scala == operator. Both datastreams must contain the key column.

    The given datastream is fully inflated when this datastream needs to be materialized. For that reason, always use the smallest datastream as the parameter, and the larger datastream as the receiver.

  52. def listener(_listener: Listener): DataStream

    Permalink
  53. val logger: Logger

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  54. def map(f: (Row) ⇒ Row): DataStream

    Permalink
  55. def mapField(fieldName: String, fn: (Any) ⇒ Any): DataStream

    Permalink
  56. def mapFieldIfExists(fieldName: String, fn: (Any) ⇒ Any): DataStream

    Permalink
  57. def maxBy[T](fn: (Row) ⇒ T)(implicit ordering: Ordering[T]): Row

    Permalink
  58. def minBy[T](fn: (Row) ⇒ T)(implicit ordering: Ordering[T]): Row

    Permalink
  59. def multiplex(count: Int): Seq[DataStream]

    Permalink
  60. final def ne(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  61. final def notify(): Unit

    Permalink
    Definition Classes
    AnyRef
  62. final def notifyAll(): Unit

    Permalink
    Definition Classes
    AnyRef
  63. def projection(fields: Seq[String]): DataStream

    Permalink

    Returns a new DataStream which contains the given list of fields from the existing stream.

  64. def projection(first: String, rest: String*): DataStream

    Permalink
  65. def projectionExpression(expr: String): DataStream

    Permalink
  66. def removeField(fieldName: String, caseSensitive: Boolean = true): DataStream

    Permalink
  67. def removeFieldIfExists(fieldName: String, caseSensitive: Boolean = true): DataStream

    Permalink
  68. def removeFields(regex: Regex): DataStream

    Permalink
  69. def renameField(nameFrom: String, nameTo: String): DataStream

    Permalink
  70. def replace(from: String, target: Any): DataStream

    Permalink
  71. def replace(fieldName: String, from: String, target: Any, errorIfUnknownField: Boolean = true): DataStream

    Permalink
  72. def replace(fieldName: String, from: String, target: Any): DataStream

    Permalink
  73. def replace(fieldName: String, fn: (Any) ⇒ Any, errorIfUnknownField: Boolean): DataStream

    Permalink
  74. def replace(fieldName: String, fn: (Any) ⇒ Any): DataStream

    Permalink
  75. def replaceField(name: String, field: Field): DataStream

    Permalink
  76. def replaceFieldType(regex: Regex, datatype: DataType): DataStream

    Permalink
  77. def replaceFieldType(from: DataType, to: DataType): DataStream

    Permalink
  78. def replaceFieldType(fieldName: String, datatype: DataType): DataStream

    Permalink

    Returns the same data but with an updated schema.

    Returns the same data but with an updated schema. The field that matches the given name will have its datatype set to the given datatype.

  79. def replaceNullValues(defaultValue: String): DataStream

    Permalink
  80. def sample(k: Int): DataStream

    Permalink

    Returns a new DataStream where only each "k" row is retained.

    Returns a new DataStream where only each "k" row is retained. Ie, if sample is 2, then on average, every other row will be returned. If sample is 10 then only 10% of rows will be returned. When running concurrently, the rows that are sampled will vary depending on the ordering that the workers pull through the rows. Each partition uses its own couter.

  81. def size: Long

    Permalink
  82. def stripCharsFromFieldNames(chars: Seq[Char]): DataStream

    Permalink

    Returns a new DataStream with the same data as this stream, but where the field names have been sanitized by removing any occurances of the given characters.

  83. def substract(stream: DataStream): DataStream

    Permalink
  84. final def synchronized[T0](arg0: ⇒ T0): T0

    Permalink
    Definition Classes
    AnyRef
  85. def take(n: Int): DataStream

    Permalink
  86. def takeWhile(fieldName: String, p: (Any) ⇒ Boolean): DataStream

    Permalink
  87. def takeWhile(p: (Row) ⇒ Boolean): DataStream

    Permalink
  88. def tee(schema: StructType, fn: (Row) ⇒ Seq[Row]): (DataStream, DataStream)

    Permalink

    Invoking this method returns two DataStreams.

    Invoking this method returns two DataStreams. The first is the original datastream which will continue as is. The second is a DataStream which is fed by rows generated from the given function. The function is invoked for each row that passes through this stream.

    Cancellation requests in the tee'd datastream do not propagate back to the original stream.

  89. def to(sink: Sink, parallelism: Int): Long

    Permalink
  90. def to(sink: Sink): Long

    Permalink
  91. def toDataTable: DataTable

    Permalink
  92. def toSet: Set[Row]

    Permalink
  93. def toString(): String

    Permalink
    Definition Classes
    AnyRef → Any
  94. def toVector: Vector[Row]

    Permalink

    Action which results in all the rows being returned in memory as a Vector.

    Action which results in all the rows being returned in memory as a Vector. Alias for 'collect()'

  95. def union(other: DataStream): DataStream

    Permalink
  96. def update(from: String, target: Any): DataStream

    Permalink

    For each row, any values that match "from" will be replaced with "target".

    For each row, any values that match "from" will be replaced with "target". This operation applies to all fields for all rows.

  97. def update(fieldName: String, from: String, target: Any, errorIfUnknownField: Boolean = true): DataStream

    Permalink

    Replaces any values that match "form" with the value "target".

    Replaces any values that match "form" with the value "target". This operation only applies to the field name specified.

    errorIfUnknownField

    throw an exception if the field specified does not exist in the dataset If set to false, then this operation will be a no-op if the field does not exist.

  98. def update(fieldName: String, from: String, target: Any): DataStream

    Permalink
  99. def update(fieldName: String, fn: (Any) ⇒ Any, errorIfUnknownField: Boolean): DataStream

    Permalink
  100. def update(fieldName: String, fn: (Any) ⇒ Any): DataStream

    Permalink

    For each row, the value corresponding to the given fieldName is applied to the function.

    For each row, the value corresponding to the given fieldName is applied to the function. The result of the function is the new value for that cell.

  101. final def wait(): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  102. final def wait(arg0: Long, arg1: Int): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  103. final def wait(arg0: Long): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  104. def withLowerCaseSchema(): DataStream

    Permalink

Deprecated Value Members

  1. def addField(name: String, fn: (Row) ⇒ Any, errorIfFieldExists: Boolean): DataStream

    Permalink
    Annotations
    @deprecated
    Deprecated

    (Since version 1.3.0) Use addFieldFn for better type inference

  2. def addField(name: String, fn: (Row) ⇒ Any): DataStream

    Permalink

    Returns a new DataStream with a new field added at the end.

    Returns a new DataStream with a new field added at the end. The datatype for the field is assumed to be String. The value for the field is taken from the function which is invoked for each row.

    Annotations
    @deprecated
    Deprecated

    (Since version 1.3.0) Use addFieldFn for better type inference

  3. def addField(field: Field, fn: (Row) ⇒ Any, errorIfFieldExists: Boolean): DataStream

    Permalink
    Annotations
    @deprecated
    Deprecated

    (Since version 1.3.0) use addFieldFn

  4. def addField(field: Field, fn: (Row) ⇒ Any): DataStream

    Permalink
    Annotations
    @deprecated
    Deprecated

    (Since version 1.3.0) use addFieldFn

  5. def addFieldIfNotExists(field: Field, defaultValue: Any): DataStream

    Permalink
    Annotations
    @deprecated
    Deprecated

    (Since version 1.3.0) use addField with errorIfFieldExists = false

  6. def addFieldIfNotExists(name: String, defaultValue: Any): DataStream

    Permalink
    Annotations
    @deprecated
    Deprecated

    (Since version 1.3.0) use addField with errorIfFieldExists = false

Inherited from Logging

Inherited from AnyRef

Inherited from Any

Ungrouped