Trait/Object

com.spotify.scio.values

SCollection

Related Docs: object SCollection | package values

Permalink

sealed trait SCollection[T] extends PCollectionWrapper[T]

A Scala wrapper for PCollection. Represents an immutable, partitioned collection of elements that can be operated on in parallel. This class contains the basic operations available on all SCollections, such as map, filter, and sum. In addition, PairSCollectionFunctions contains operations available only on SCollections of key-value pairs, such as groupByKey and join; DoubleSCollectionFunctions contains operations available only on SCollections of Doubles.

Linear Supertypes
PCollectionWrapper[T], TransformNameable, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. SCollection
  2. PCollectionWrapper
  3. TransformNameable
  4. AnyRef
  5. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Abstract Value Members

  1. abstract val context: ScioContext

    Permalink

    The ScioContext associated with this PCollection.

    The ScioContext associated with this PCollection.

    Definition Classes
    PCollectionWrapper
  2. implicit abstract val ct: ClassTag[T]

    Permalink
    Definition Classes
    PCollectionWrapper
  3. abstract val internal: PCollection[T]

    Permalink

    The PCollection being wrapped internally.

    The PCollection being wrapped internally.

    Definition Classes
    PCollectionWrapper

Concrete Value Members

  1. final def !=(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int

    Permalink
    Definition Classes
    AnyRef → Any
  3. def ++(that: SCollection[T]): SCollection[T]

    Permalink

    Return the union of this SCollection and another one.

    Return the union of this SCollection and another one. Any identical elements will appear multiple times (use distinct to eliminate them).

  4. final def ==(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  5. def aggregate[A, U](aggregator: Aggregator[T, A, U])(implicit arg0: ClassTag[A], arg1: ClassTag[U]): SCollection[U]

    Permalink

    Aggregate with Aggregator.

    Aggregate with Aggregator. First each item T is mapped to A, then we reduce with a Semigroup of A, then finally we present the results as U. This could be more powerful and better optimized in some cases.

  6. def aggregate[U](zeroValue: U)(seqOp: (U, T) ⇒ U, combOp: (U, U) ⇒ U)(implicit arg0: ClassTag[U]): SCollection[U]

    Permalink

    Aggregate the elements using given combine functions and a neutral "zero value".

    Aggregate the elements using given combine functions and a neutral "zero value". This function can return a different result type, U, than the type of this SCollection, T. Thus, we need one operation for merging a T into an U and one operation for merging two U's. Both of these functions are allowed to modify and return their first argument instead of creating a new U to avoid memory allocation.

  7. def applyKvTransform[K, V](transform: PTransform[_ >: PCollection[T], PCollection[KV[K, V]]])(implicit arg0: ClassTag[K], arg1: ClassTag[V]): SCollection[KV[K, V]]

    Permalink

    Apply a PTransform and wrap the output in an SCollection.

    Apply a PTransform and wrap the output in an SCollection. This is a special case of applyTransform for transforms with KV output.

  8. def applyTransform[U](transform: PTransform[_ >: PCollection[T], PCollection[U]])(implicit arg0: ClassTag[U]): SCollection[U]

    Permalink

    Apply a PTransform and wrap the output in an SCollection.

  9. final def asInstanceOf[T0]: T0

    Permalink
    Definition Classes
    Any
  10. def asIterableSideInput: SideInput[Iterable[T]]

    Permalink

    Convert this SCollection to a SideInput, mapping each window to an Iterable, to be used with withSideInputs.

    Convert this SCollection to a SideInput, mapping each window to an Iterable, to be used with withSideInputs.

    The values of the Iterable for a window are not required to fit in memory, but they may also not be effectively cached. If it is known that every window fits in memory, and stronger caching is desired, use asListSideInput.

  11. def asListSideInput: SideInput[Seq[T]]

    Permalink

    Convert this SCollection to a SideInput, mapping each window to a Seq, to be used with withSideInputs.

    Convert this SCollection to a SideInput, mapping each window to a Seq, to be used with withSideInputs.

    The resulting Seq is required to fit in memory.

  12. def asSingletonSideInput(defaultValue: T): SideInput[T]

    Permalink

    Convert this SCollection of a single value per window to a SideInput with a default value, to be used with withSideInputs.

  13. def asSingletonSideInput: SideInput[T]

    Permalink

    Convert this SCollection of a single value per window to a SideInput, to be used with withSideInputs.

  14. def clone(): AnyRef

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  15. def collect[U](pfn: PartialFunction[T, U])(implicit arg0: ClassTag[U]): SCollection[U]

    Permalink

    Filter the elements for which the given PartialFunction is defined, and then map.

  16. def combine[C](createCombiner: (T) ⇒ C)(mergeValue: (C, T) ⇒ C)(mergeCombiners: (C, C) ⇒ C)(implicit arg0: ClassTag[C]): SCollection[C]

    Permalink

    Generic function to combine the elements using a custom set of aggregation functions.

    Generic function to combine the elements using a custom set of aggregation functions. Turns an SCollection[T] into a result of type SCollection[C], for a "combined type" C. Note that V and C can be different -- for example, one might combine an SCollection of type Int into an SCollection of type Seq[Int]. Users provide three functions:

    - createCombiner, which turns a V into a C (e.g., creates a one-element list)

    - mergeValue, to merge a V into a C (e.g., adds it to the end of a list)

    - mergeCombiners, to combine two C's into a single one.

  17. def count: SCollection[Long]

    Permalink

    Count the number of elements in the SCollection.

    Count the number of elements in the SCollection.

    returns

    a new SCollection with the count

  18. def countApproxDistinct(maximumEstimationError: Double = 0.02): SCollection[Long]

    Permalink

    Count approximate number of distinct elements in the SCollection.

    Count approximate number of distinct elements in the SCollection.

    maximumEstimationError

    the maximum estimation error, which should be in the range [0.01, 0.5]

  19. def countApproxDistinct(sampleSize: Int): SCollection[Long]

    Permalink

    Count approximate number of distinct elements in the SCollection.

    Count approximate number of distinct elements in the SCollection.

    sampleSize

    the number of entries in the statistical sample; the higher this number, the more accurate the estimate will be; should be >= 16

  20. def countByValue: SCollection[(T, Long)]

    Permalink

    Count of each unique value in this SCollection as an SCollection of (value, count) pairs.

  21. def cross[U](that: SCollection[U])(implicit arg0: ClassTag[U]): SCollection[(T, U)]

    Permalink

    Return the cross product with another SCollection by replicating that to all workers.

    Return the cross product with another SCollection by replicating that to all workers. The right side should be tiny and fit in memory.

  22. def debug(out: () ⇒ PrintStream = () => Console.out, prefix: String = "", enabled: Boolean = true): SCollection[T]

    Permalink

    Print content of an SCollection to out().

    Print content of an SCollection to out().

    out

    where to write the debug information. Default: stdout

    prefix

    prefix for each logged entry. Default: empty string

    enabled

    if debugging is enabled or not. Default: true. It can be useful to set this to sc.isTest to avoid debugging when running in production.

  23. def distinct: SCollection[T]

    Permalink

    Return a new SCollection containing the distinct elements in this SCollection.

  24. final def eq(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  25. def equals(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  26. def filter(f: (T) ⇒ Boolean): SCollection[T]

    Permalink

    Return a new SCollection containing only the elements that satisfy a predicate.

  27. def finalize(): Unit

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  28. def flatMap[U](f: (T) ⇒ TraversableOnce[U])(implicit arg0: ClassTag[U]): SCollection[U]

    Permalink

    Return a new SCollection by first applying a function to all elements of this SCollection, and then flattening the results.

  29. def flatten[U](implicit arg0: ClassTag[U], ev: (T) ⇒ TraversableOnce[U]): SCollection[U]

    Permalink

    Return a new SCollection[U] by flattening each element of an SCollection[Traversable[U]].

  30. def fold(implicit mon: Monoid[T]): SCollection[T]

    Permalink

    Fold with Monoid, which defines the associative function and "zero value" for T.

    Fold with Monoid, which defines the associative function and "zero value" for T. This could be more powerful and better optimized in some cases.

  31. def fold(zeroValue: T)(op: (T, T) ⇒ T): SCollection[T]

    Permalink

    Aggregate the elements using a given associative function and a neutral "zero value".

    Aggregate the elements using a given associative function and a neutral "zero value". The function op(t1, t2) is allowed to modify t1 and return it as its result value to avoid object allocation; however, it should not modify t2.

  32. final def getClass(): Class[_]

    Permalink
    Definition Classes
    AnyRef → Any
  33. def groupBy[K](f: (T) ⇒ K)(implicit arg0: ClassTag[K]): SCollection[(K, Iterable[T])]

    Permalink

    Return an SCollection of grouped items.

    Return an SCollection of grouped items. Each group consists of a key and a sequence of elements mapping to that key. The ordering of elements within each group is not guaranteed, and may even differ each time the resulting SCollection is evaluated.

    Note: This operation may be very expensive. If you are grouping in order to perform an aggregation (such as a sum or average) over each key, using PairSCollectionFunctions.aggregateByKey or PairSCollectionFunctions.reduceByKey will provide much better performance.

  34. def hashCode(): Int

    Permalink
    Definition Classes
    AnyRef → Any
  35. def hashLookup[V](that: SCollection[(T, V)])(implicit arg0: ClassTag[V]): SCollection[(T, Iterable[V])]

    Permalink

    Look up values in an SCollection[(T, V)] for each element T in this SCollection by replicating that to all workers.

    Look up values in an SCollection[(T, V)] for each element T in this SCollection by replicating that to all workers. The right side should be tiny and fit in memory.

  36. def intersection(that: SCollection[T]): SCollection[T]

    Permalink

    Return the intersection of this SCollection and another one.

    Return the intersection of this SCollection and another one. The output will not contain any duplicate elements, even if the input SCollections did.

    Note that this method performs a shuffle internally.

  37. final def isInstanceOf[T0]: Boolean

    Permalink
    Definition Classes
    Any
  38. def keyBy[K](f: (T) ⇒ K)(implicit arg0: ClassTag[K]): SCollection[(K, T)]

    Permalink

    Create tuples of the elements in this SCollection by applying f.

  39. def map[U](f: (T) ⇒ U)(implicit arg0: ClassTag[U]): SCollection[U]

    Permalink

    Return a new SCollection by applying a function to all elements of this SCollection.

  40. def materialize: Future[Tap[T]]

    Permalink

    Extract data from this SCollection as a Future.

    Extract data from this SCollection as a Future. The Future will be completed once the pipeline completes successfully.

  41. def max(implicit ord: Ordering[T]): SCollection[T]

    Permalink

    Return the max of this SCollection as defined by the implicit Ordering[T].

    Return the max of this SCollection as defined by the implicit Ordering[T].

    returns

    a new SCollection with the maximum element

  42. def mean(implicit ev: Numeric[T]): SCollection[Double]

    Permalink

    Return the mean of this SCollection as defined by the implicit Numeric[T].

    Return the mean of this SCollection as defined by the implicit Numeric[T].

    returns

    a new SCollection with the mean of elements

  43. def min(implicit ord: Ordering[T]): SCollection[T]

    Permalink

    Return the min of this SCollection as defined by the implicit Ordering[T].

    Return the min of this SCollection as defined by the implicit Ordering[T].

    returns

    a new SCollection with the minimum element

  44. def name: String

    Permalink

    A friendly name for this SCollection.

  45. final def ne(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  46. final def notify(): Unit

    Permalink
    Definition Classes
    AnyRef
  47. final def notifyAll(): Unit

    Permalink
    Definition Classes
    AnyRef
  48. def pApply[U](transform: PTransform[_ >: PCollection[T], PCollection[U]])(implicit arg0: ClassTag[U]): SCollection[U]

    Permalink
    Attributes
    protected
    Definition Classes
    PCollectionWrapper
  49. def partition(numPartitions: Int, f: (T) ⇒ Int): Seq[SCollection[T]]

    Permalink

    Partition this SCollection with the provided function.

    Partition this SCollection with the provided function.

    numPartitions

    number of output partitions

    f

    function that assigns an output partition to each element, should be in the range [0, numPartitions - 1]

    returns

    partitioned SCollections in a Seq

  50. def quantilesApprox(numQuantiles: Int)(implicit ord: Ordering[T]): SCollection[Iterable[T]]

    Permalink

    Compute the SCollection's data distribution using approximate N-tiles.

    Compute the SCollection's data distribution using approximate N-tiles.

    returns

    a new SCollection whose single value is an Iterable of the approximate N-tiles of the elements

  51. def randomSplit(weightA: Double, weightB: Double): (SCollection[T], SCollection[T], SCollection[T])

    Permalink

    Randomly splits this SCollection into three parts.

    Randomly splits this SCollection into three parts. Note: 0 < weightA + weightB < 1

    weightA

    weight for first SCollection, should be in the range (0, 1)

    weightB

    weight for second SCollection, should be in the range (0, 1)

    returns

    split SCollections in a Tuple3

  52. def randomSplit(weight: Double): (SCollection[T], SCollection[T])

    Permalink

    Randomly splits this SCollection into two parts.

    Randomly splits this SCollection into two parts.

    weight

    weight for left hand side SCollection, should be in the range (0, 1)

    returns

    split SCollections in a Tuple2

  53. def randomSplit(weights: Array[Double]): Array[SCollection[T]]

    Permalink

    Randomly splits this SCollection with the provided weights.

    Randomly splits this SCollection with the provided weights.

    weights

    weights for splits, will be normalized if they don't sum to 1

    returns

    split SCollections in an array

  54. def reduce(op: (T, T) ⇒ T): SCollection[T]

    Permalink

    Reduce the elements of this SCollection using the specified commutative and associative binary operator.

  55. def sample(withReplacement: Boolean, fraction: Double): SCollection[T]

    Permalink

    Return a sampled subset of this SCollection.

  56. def sample(sampleSize: Int): SCollection[Iterable[T]]

    Permalink

    Return a sampled subset of this SCollection.

    Return a sampled subset of this SCollection.

    returns

    a new SCollection whose single value is an Iterable of the samples

  57. def saveAsAvroFile(path: String, numShards: Int = 0, schema: Schema = null, suffix: String = "", codec: CodecFactory = CodecFactory.deflateCodec(6), metadata: Map[String, AnyRef] = Map.empty): Future[Tap[T]]

    Permalink

    Save this SCollection as an Avro file.

    Save this SCollection as an Avro file.

    schema

    must be not null if T is of type GenericRecord.

  58. def saveAsBigQuery(tableSpec: String, schema: TableSchema = null, writeDisposition: WriteDisposition = null, createDisposition: CreateDisposition = null, tableDescription: String = null)(implicit ev: <:<[T, TableRow]): Future[Tap[TableRow]]

    Permalink

    Save this SCollection as a BigQuery table.

    Save this SCollection as a BigQuery table. Note that elements must be of type TableRow.

  59. def saveAsBigQuery(table: TableReference, schema: TableSchema, writeDisposition: WriteDisposition, createDisposition: CreateDisposition, tableDescription: String)(implicit ev: <:<[T, TableRow]): Future[Tap[TableRow]]

    Permalink

    Save this SCollection as a BigQuery table.

    Save this SCollection as a BigQuery table. Note that elements must be of type TableRow.

  60. def saveAsCustomOutput[O <: POutput](name: String, transform: PTransform[PCollection[T], O]): Future[Tap[T]]

    Permalink

    Save this SCollection with a custom output transform.

    Save this SCollection with a custom output transform. The transform should have a unique name.

  61. def saveAsDatastore(projectId: String)(implicit ev: <:<[T, Entity]): Future[Tap[Entity]]

    Permalink

    Save this SCollection as a Datastore dataset.

    Save this SCollection as a Datastore dataset. Note that elements must be of type Entity.

  62. def saveAsObjectFile(path: String, numShards: Int = 0, suffix: String = ".obj", metadata: Map[String, AnyRef] = Map.empty): Future[Tap[T]]

    Permalink

    Save this SCollection as an object file using default serialization.

    Save this SCollection as an object file using default serialization.

    Serialized objects are stored in Avro files to leverage Avro's block file format. Note that serialization is not guaranteed to be compatible across Scio releases.

  63. def saveAsProtobufFile(path: String, numShards: Int = 0)(implicit ev: <:<[T, Message]): Future[Tap[T]]

    Permalink

    Save this SCollection as a Protobuf file.

    Save this SCollection as a Protobuf file.

    Protobuf messages are serialized into Array[Byte] and stored in Avro files to leverage Avro's block file format.

  64. def saveAsPubsub(topic: String, idAttribute: String = null, timestampAttribute: String = null): Future[Tap[T]]

    Permalink

    Save this SCollection as a Pub/Sub topic.

  65. def saveAsPubsubWithAttributes[V](topic: String, idAttribute: String = null, timestampAttribute: String = null)(implicit arg0: ClassTag[V], ev: <:<[T, (V, Map[String, String])]): Future[Tap[V]]

    Permalink

    Save this SCollection as a Pub/Sub topic using the given map as message attributes.

  66. def saveAsTableRowJsonFile(path: String, numShards: Int = 0, compression: Compression = Compression.UNCOMPRESSED)(implicit ev: <:<[T, TableRow]): Future[Tap[TableRow]]

    Permalink

    Save this SCollection as a BigQuery TableRow JSON text file.

    Save this SCollection as a BigQuery TableRow JSON text file. Note that elements must be of type TableRow.

  67. def saveAsTextFile(path: String, suffix: String = ".txt", numShards: Int = 0, compression: Compression = Compression.UNCOMPRESSED): Future[Tap[String]]

    Permalink

    Save this SCollection as a text file.

    Save this SCollection as a text file. Note that elements must be of type String.

  68. def saveAsTypedAvroFile(path: String, numShards: Int = 0, suffix: String = "", codec: CodecFactory = CodecFactory.deflateCodec(6), metadata: Map[String, AnyRef] = Map.empty)(implicit ct: ClassTag[T], tt: scala.reflect.api.JavaUniverse.TypeTag[T], ev: <:<[T, HasAvroAnnotation]): Future[Tap[T]]

    Permalink

    Save this SCollection as an Avro file.

    Save this SCollection as an Avro file. Note that element type T must be a case class annotated with AvroType.toSchema.

  69. def saveAsTypedBigQuery(tableSpec: String, writeDisposition: WriteDisposition = null, createDisposition: CreateDisposition = null)(implicit ct: ClassTag[T], tt: scala.reflect.api.JavaUniverse.TypeTag[T], ev: <:<[T, HasAnnotation]): Future[Tap[T]]

    Permalink

    Save this SCollection as a BigQuery table.

    Save this SCollection as a BigQuery table. Note that element type T must be annotated with BigQueryType.

    This could be a complete case class with BigQueryType.toTable. For example:

    @BigQueryType.toTable
    case class Result(name: String, score: Double)
    
    val p: SCollection[Result] = // process data and convert elements to Result
    p.saveAsTypedBigQuery("myproject:mydataset.mytable")

    It could also be an empty class with schema from BigQueryType.fromSchema, BigQueryType.fromTable, or BigQueryType.fromQuery. For example:

    @BigQueryType.fromTable("publicdata:samples.gsod")
    class Row
    
    sc.typedBigQuery[Row]()
      .sample(withReplacement = false, fraction = 0.1)
      .saveAsTypedBigQuery("myproject:samples.gsod")
  70. def saveAsTypedBigQuery(table: TableReference, writeDisposition: WriteDisposition, createDisposition: CreateDisposition)(implicit ct: ClassTag[T], tt: scala.reflect.api.JavaUniverse.TypeTag[T], ev: <:<[T, HasAnnotation]): Future[Tap[T]]

    Permalink

    Save this SCollection as a BigQuery table.

    Save this SCollection as a BigQuery table. Note that element type T must be a case class annotated with BigQueryType.toTable.

  71. def setCoder(coder: Coder[T]): SCollection[T]

    Permalink

    Assign a Coder to this SCollection.

  72. def setName(name: String): SCollection[T]

    Permalink

    Assign a name to this SCollection.

  73. def subtract(that: SCollection[T]): SCollection[T]

    Permalink

    Return an SCollection with the elements from this that are not in other.

  74. def sum(implicit sg: Semigroup[T]): SCollection[T]

    Permalink

    Reduce with Semigroup.

    Reduce with Semigroup. This could be more powerful and better optimized than reduce in some cases.

  75. final def synchronized[T0](arg0: ⇒ T0): T0

    Permalink
    Definition Classes
    AnyRef
  76. def take(num: Long): SCollection[T]

    Permalink

    Return a sampled subset of any num elements of the SCollection.

  77. def timestampBy(f: (T) ⇒ Instant, allowedTimestampSkew: Duration = Duration.ZERO): SCollection[T]

    Permalink

    Assign timestamps to values.

    Assign timestamps to values. With a optional skew

  78. def toString(): String

    Permalink
    Definition Classes
    AnyRef → Any
  79. def toWindowed: WindowedSCollection[T]

    Permalink

    Convert this SCollection to an WindowedSCollection.

  80. def top(num: Int)(implicit ord: Ordering[T]): SCollection[Iterable[T]]

    Permalink

    Return the top k (largest) elements from this SCollection as defined by the specified implicit Ordering[T].

    Return the top k (largest) elements from this SCollection as defined by the specified implicit Ordering[T].

    returns

    a new SCollection whose single value is an Iterable of the top k

  81. def union(that: SCollection[T]): SCollection[T]

    Permalink

    Return the union of this SCollection and another one.

    Return the union of this SCollection and another one. Any identical elements will appear multiple times (use distinct to eliminate them).

  82. final def wait(): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  83. final def wait(arg0: Long, arg1: Int): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  84. final def wait(arg0: Long): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  85. def windowByDays(number: Int, options: WindowOptions = WindowOptions()): SCollection[T]

    Permalink

    Window values into by days.

  86. def windowByMonths(number: Int, options: WindowOptions = WindowOptions()): SCollection[T]

    Permalink

    Window values into by months.

  87. def windowByWeeks(number: Int, startDayOfWeek: Int, options: WindowOptions = WindowOptions()): SCollection[T]

    Permalink

    Window values into by weeks.

  88. def windowByYears(number: Int, options: WindowOptions = WindowOptions()): SCollection[T]

    Permalink

    Window values into by years.

  89. def withFanout(fanout: Int): SCollectionWithFanout[T]

    Permalink

    Convert this SCollection to an SCollectionWithFanout that uses an intermediate node to combine parts of the data to reduce load on the final global combine step.

    Convert this SCollection to an SCollectionWithFanout that uses an intermediate node to combine parts of the data to reduce load on the final global combine step.

    fanout

    the number of intermediate keys that will be used

  90. def withFixedWindows(duration: Duration, offset: Duration = Duration.ZERO, options: WindowOptions = WindowOptions()): SCollection[T]

    Permalink

    Window values into fixed windows.

  91. def withGlobalWindow(options: WindowOptions = WindowOptions()): SCollection[T]

    Permalink

    Group values in to a single global window.

  92. def withName(name: String): SCollection.this.type

    Permalink

    Set a custom name for the next transform to be applied.

    Set a custom name for the next transform to be applied.

    Definition Classes
    TransformNameable
  93. def withPaneInfo: SCollection[(T, PaneInfo)]

    Permalink

    Convert values into pairs of (value, window).

  94. def withSessionWindows(gapDuration: Duration, options: WindowOptions = WindowOptions()): SCollection[T]

    Permalink

    Window values based on sessions.

  95. def withSideInputs(sides: SideInput[_]*): SCollectionWithSideInput[T]

    Permalink

    Convert this SCollection to an SCollectionWithSideInput with one or more SideInputs, similar to Spark broadcast variables.

    Convert this SCollection to an SCollectionWithSideInput with one or more SideInputs, similar to Spark broadcast variables. Call SCollectionWithSideInput.toSCollection when done with side inputs.

    val s1: SCollection[Int] = // ...
    val s2: SCollection[String] = // ...
    val s3: SCollection[(String, Double)] = // ...
    
    // Prepare side inputs
    val side1 = s1.asSingletonSideInput
    val side2 = s2.asIterableSideInput
    val side3 = s3.asMapSideInput
    
    val p: SCollection[MyRecord] = // ...
    p.withSideInputs(side1, side2, side3).map { (x, s) =>
      // Extract side inputs from context
      val s1: Int = s(side1)
      val s2: Iterable[String] = s(side2)
      val s3: Map[String, Iterable[Double]] = s(side3)
      // ...
    }
  96. def withSideOutputs(sides: SideOutput[_]*): SCollectionWithSideOutput[T]

    Permalink

    Convert this SCollection to an SCollectionWithSideOutput with one or more SideOutputs, so that a single transform can write to multiple destinations.

    Convert this SCollection to an SCollectionWithSideOutput with one or more SideOutputs, so that a single transform can write to multiple destinations.

    // Prepare side inputs
    val side1 = SideOutput[String]()
    val side2 = SideOutput[Int]()
    
    val p: SCollection[MyRecord] = // ...
    p.withSideOutputs(side1, side2).map { (x, s) =>
      // Write to side outputs via context
      s.output(side1, "word").output(side2, 1)
      // ...
    }
  97. def withSlidingWindows(size: Duration, period: Duration = null, offset: Duration = Duration.ZERO, options: WindowOptions = WindowOptions()): SCollection[T]

    Permalink

    Window values into sliding windows.

  98. def withTimestamp: SCollection[(T, Instant)]

    Permalink

    Convert values into pairs of (value, timestamp).

  99. def withWindow[W <: BoundedWindow]: SCollection[(T, W)]

    Permalink

    Convert values into pairs of (value, window).

    Convert values into pairs of (value, window).

    W

    window type, must be BoundedWindow or one of it's sub-types, e.g. GlobalWindow if this SCollection is not windowed or IntervalWindow if it is windowed.

  100. def withWindowFn[W <: BoundedWindow](fn: WindowFn[AnyRef, W], options: WindowOptions = WindowOptions()): SCollection[T]

    Permalink

    Window values with the given function.

Inherited from PCollectionWrapper[T]

Inherited from TransformNameable

Inherited from AnyRef

Inherited from Any

Collection Operations

debug

Hash Operations

Output Sinks

Side Input and Output Operations

Transformations

Windowing Operations

Ungrouped