The ScioContext associated with this PCollection.
The ScioContext associated with this PCollection.
The PCollection being wrapped internally.
The PCollection being wrapped internally.
Return the union of this SCollection and another one.
Return the union of this SCollection and another one. Any identical elements will appear
multiple times (use .distinct() to eliminate them).
Aggregate with Aggregator.
Aggregate with Aggregator. First each item T is mapped to A, then we reduce with a semigroup of A, then finally we present the results as U. This could be more powerful and better optimized in some cases.
Aggregate the elements using given combine functions and a neutral "zero value".
Aggregate the elements using given combine functions and a neutral "zero value". This function can return a different result type, U, than the type of this SCollection, T. Thus, we need one operation for merging a T into an U and one operation for merging two U's. Both of these functions are allowed to modify and return their first argument instead of creating a new U to avoid memory allocation.
Apply a PTransform and wrap the output in an SCollection.
Convert this SCollection to a SideInput, mapping each window to an Iterable, to be used with SCollection.withSideInputs.
Convert this SCollection to a SideInput, mapping each window to an Iterable, to be used with SCollection.withSideInputs.
The values of the Iterable for a window are not required to fit in memory, but they may also not be effectively cached. If it is known that every window fits in memory, and stronger caching is desired, use asListSideInput.
Convert this SCollection to a SideInput, mapping each window to a List, to be used with SCollection.withSideInputs.
Convert this SCollection of a single value per window to a SideInput, to be used with SCollection.withSideInputs.
Filter the elements for which the given PartialFunction is defined, and then map.
Generic function to combine the elements using a custom set of aggregation functions.
Generic function to combine the elements using a custom set of aggregation functions. Turns an SCollection[T] into a result of type SCollection[C], for a "combined type" C. Note that V and C can be different -- for example, one might combine an SCollection of type Int into an SCollection of type Seq[Int]. Users provide three functions:
- createCombiner, which turns a V into a C (e.g., creates a one-element list)
- mergeValue, to merge a V into a C (e.g., adds it to the end of a list)
- mergeCombiners, to combine two C's into a single one.
Count the number of elements in the SCollection.
Count the number of elements in the SCollection.
a new SCollection with the count
Count approximate number of distinct elements in the SCollection.
Count approximate number of distinct elements in the SCollection.
the maximum estimation error, which should be in the range
[0.01, 0.5]
Count approximate number of distinct elements in the SCollection.
Count approximate number of distinct elements in the SCollection.
the number of entries in the statisticalsample; the higher this number, the
more accurate the estimate will be; should be >= 16
Count of each unique value in this SCollection as an SCollection of (value, count) pairs.
Return the cross product with another SCollection by replicating that to all workers.
Return the cross product with another SCollection by replicating that to all workers. The
right side should be tiny and fit in memory.
Print content of a SCollection to out().
Return a new SCollection containing the distinct elements in this SCollection.
Return a new SCollection containing only the elements that satisfy a predicate.
Return a new SCollection by first applying a function to all elements of this SCollection, and then flattening the results.
Fold with Monoid, which defines the associative function and "zero value" for T.
Fold with Monoid, which defines the associative function and "zero value" for T. This could be more powerful and better optimized in some cases.
Aggregate the elements using a given associative function and a neutral "zero value".
Aggregate the elements using a given associative function and a neutral "zero value". The function op(t1, t2) is allowed to modify t1 and return it as its result value to avoid object allocation; however, it should not modify t2.
Return an SCollection of grouped items.
Return an SCollection of grouped items. Each group consists of a key and a sequence of elements mapping to that key. The ordering of elements within each group is not guaranteed, and may even differ each time the resulting SCollection is evaluated.
Note: This operation may be very expensive. If you are grouping in order to perform an aggregation (such as a sum or average) over each key, using PairSCollectionFunctions.aggregateByKey or PairSCollectionFunctions.reduceByKey will provide much better performance.
Look up values in a SCollection[(T, V)] for each element T in this SCollection by replicating
that to all workers.
Look up values in a SCollection[(T, V)] for each element T in this SCollection by replicating
that to all workers. The right side should be tiny and fit in memory.
Return the intersection of this SCollection and another one.
Return the intersection of this SCollection and another one. The output will not contain any duplicate elements, even if the input SCollections did.
Note that this method performs a shuffle internally.
Create tuples of the elements in this SCollection by applying f.
Return a new SCollection by applying a function to all elements of this SCollection.
Extract data from this SCollection as a Future.
Extract data from this SCollection as a Future. The Future will be completed once the pipeline completes successfully.
Return the max of this SCollection as defined by the implicit Ordering[T].
Return the max of this SCollection as defined by the implicit Ordering[T].
a new SCollection with the maximum element
Return the mean of this SCollection as defined by the implicit Numeric[T].
Return the mean of this SCollection as defined by the implicit Numeric[T].
a new SCollection with the mean of elements
Return the min of this SCollection as defined by the implicit Ordering[T].
Return the min of this SCollection as defined by the implicit Ordering[T].
a new SCollection with the minimum element
A friendly name for this SCollection.
Partition this SCollection with the provided function.
Partition this SCollection with the provided function.
number of output partitions
function that assigns an output partition to each element, should be in the range
[0, numPartitions - 1]
partitioned SCollections in a Seq
Compute the SCollection's data distribution using approximate N-tiles.
Compute the SCollection's data distribution using approximate N-tiles.
a new SCollection whose single value is an Iterable of the approximate N-tiles of
the elements
Randomly splits this SCollection with the provided weights.
Randomly splits this SCollection with the provided weights.
weights for splits, will be normalized if they don't sum to 1
split SCollections in an array
Reduce the elements of this SCollection using the specified commutative and associative binary operator.
Return a sampled subset of this SCollection.
Return a sampled subset of this SCollection.
Return a sampled subset of this SCollection.
a new SCollection whose single value is an Iterable of the samples
Save this SCollection as an Avro file.
Save this SCollection as an Avro file.
must be not null if T is of type GenericRecord.
Save this SCollection as a BigQuery table.
Save this SCollection as a BigQuery table. Note that elements must be of type TableRow.
Save this SCollection as a BigQuery table.
Save this SCollection as a BigQuery table. Note that elements must be of type TableRow.
Save this SCollection with a custom output transform.
Save this SCollection with a custom output transform. The transform should have a unique name.
Save this SCollection as a Datastore dataset.
Save this SCollection as a Datastore dataset. Note that elements must be of type Entity.
Save this SCollection as an object file using default serialization.
Save this SCollection as a Protobuf file.
Save this SCollection as a Pub/Sub topic.
Save this SCollection as a JSON text file.
Save this SCollection as a JSON text file. Note that elements must be of type TableRow.
Save this SCollection as a text file.
Save this SCollection as a text file. Note that elements must be of type String.
Save this SCollection as a TensorFlow TFRecord file.
Save this SCollection as a TensorFlow TFRecord file. Note that elements must be of type ArrayByte.
Assign a Coder to this SCollection.
Assign a name to this SCollection.
Return an SCollection with the elements from this that are not in other.
Reduce with Semigroup.
Reduce with Semigroup. This could be more powerful and better optimized in some cases.
Return a sampled subset of any num elements of the SCollection.
Assign timestamps to values.
Assign timestamps to values. With a optional skew
Convert this SCollection to an WindowedSCollection.
Return the top k (largest) elements from this SCollection as defined by the specified implicit Ordering[T].
Return the top k (largest) elements from this SCollection as defined by the specified implicit Ordering[T].
a new SCollection whose single value is an Iterable of the top k
Return the union of this SCollection and another one.
Return the union of this SCollection and another one. Any identical elements will appear
multiple times (use .distinct() to eliminate them).
Window values into by days.
Window values into by months.
Window values into by weeks.
Window values into by years.
Convert this SCollection to an SCollectionWithAccumulator with one or more Accumulators, similar to Hadoop counters.
Convert this SCollection to an SCollectionWithAccumulator with one or more Accumulators, similar to Hadoop counters. Call SCollectionWithAccumulator.toSCollection when done with accumulators.
Note that each accumulator may be used in a single scope only.
Create accumulators with ScioContext.maxAccumulator, ScioContext.minAccumulator or ScioContext.sumAccumulator. For example:
val maxLineLength = sc.maxAccumulator[Int]("maxLineLength") val minLineLength = sc.maxAccumulator[Int]("maxLineLength") val emptyLines = sc.maxAccumulator[Long]("emptyLines") val p: SCollection[String] = // ... p .withAccumulators(maxLineLength, minLineLength, emptyLines) .filter { (l, c) => val t = l.strip() c.addValue(maxLineLength, t.length).addValue(minLineLength, t.length) val b = t.isEmpty if (b) c.addValue(emptyLines, 1L) !b } .toSCollection
Convert this SCollection to an SCollectionWithFanout that uses an intermediate node to combine parts of the data to reduce load on the final global combine step.
Convert this SCollection to an SCollectionWithFanout that uses an intermediate node to combine parts of the data to reduce load on the final global combine step.
the number of intermediate keys that will be used
Window values into fixed windows.
Group values in to a single global window.
Convert values into pairs of (value, window).
Window values based on sessions.
Convert this SCollection to an SCollectionWithSideInput with one or more SideInputs, similar to Spark broadcast variables.
Convert this SCollection to an SCollectionWithSideInput with one or more SideInputs, similar to Spark broadcast variables. Call SCollectionWithSideInput.toSCollection when done with side inputs.
Note that the side inputs should be tiny and fit in memory.
val s1: SCollection[Int] = // ... val s2: SCollection[String] = // ... val s3: SCollection[(String, Double)] = // ... // Prepare side inputs val side1 = s1.asSingletonSideInput val side2 = s2.asIterableSideInput val side3 = s3.asMapSideInput val p: SCollection[MyRecord] = // ... p.withSideInputs(side1, side2, side3).map { (x, s) => // Extract side inputs from context val s1: Int = s(side1) val s2: Iterable[String] = s(side2) val s3: Map[String, Iterable[Double]] = s(side3) // ... }
Convert this SCollection to an SCollectionWithSideOutput with one or more SideOutputs, so that a single transform can write to multiple destinations.
Convert this SCollection to an SCollectionWithSideOutput with one or more SideOutputs, so that a single transform can write to multiple destinations.
// Prepare side inputs val side1 = SideOutput[String]() val side2 = SideOutput[Int]() val p: SCollection[MyRecord] = // ... p.withSideOutputs(side1, side2).map { (x, s) => // Write to side outputs via context s.output(side1, "word").output(side2, 1) // ... }
Window values into sliding windows.
Convert values into pairs of (value, timestamp).
Convert values into pairs of (value, window).
Window values with the given function.
A Scala wrapper for PCollection. Represents an immutable, partitioned collection of elements that can be operated on in parallel. This class contains the basic operations available on all SCollections, such as
map,filter, andpersist. In addition, PairSCollectionFunctions contains operations available only on SCollections of key-value pairs, such asgroupByKeyandjoin; DoubleSCollectionFunctions contains operations available only on SCollections of Doubles.