Broadcast
object Broadcast
Provides mechanisms for broadcast distribution of elements to multiple streams.
class Object
trait Matchable
class Any
Value members
Methods
def apply[F <: ([_$1] =>> Any), O](minReady: Int)(evidence$1: Concurrent[F]): (F, O) => Stream[F, O]
Allows elements of a stream to be broadcast to multiple workers.
As the elements arrive, they are broadcast to all
element was pulled.
workers that have started evaluation before theelement was pulled.
Elements are pulled as chunks from the source and the next chunk is pulled when all workers are done
with processing the current chunk. This behaviour may slow down processing of incoming chunks by
faster workers. If this is not desired, consider using the
for slower workers.
with processing the current chunk. This behaviour may slow down processing of incoming chunks by
faster workers. If this is not desired, consider using the
prefetch and prefetchN combinators on workers to compensatefor slower workers.
Often this combinator is used together with parJoin, such as:
{{{
Stream(1,2,3,4).covary[IO] .broadcast.zipWithIndex.map { case (worker, idx) =>
worker.evalMap { o => IO(println(s"$idx: $o")) }
}.take(3).parJoinUnbounded.compile.drain.unsafeRunSync()
}}}
Stream(1,2,3,4).covary[IO] .broadcast.zipWithIndex.map { case (worker, idx) =>
worker.evalMap { o => IO(println(s"$idx: $o")) }
}.take(3).parJoinUnbounded.compile.drain.unsafeRunSync()
}}}
Note that in the example, above the workers are not guaranteed to see all elements emitted. This is
due to different subscription times of each worker and speed of emitting the elements by the source.
If this is not desired, consider using
They will hold on pulling from source if there are no workers ready.
due to different subscription times of each worker and speed of emitting the elements by the source.
If this is not desired, consider using
broacastThrough and broadcastTo, which are built on top of Broadcast.through, as an alternative.They will hold on pulling from source if there are no workers ready.
When
from
resulting stream will not terminate until the inner streams terminate.
source terminates, then the inner streams (workers) are terminated once all elements pulledfrom
source are processed by all workers. However, note that when that source terminates,resulting stream will not terminate until the inner streams terminate.
- Value Params
- minReady
-
specifies that broadcasting will hold off until at least
minReadysubscribers will
be ready
def through[F <: ([_$2] =>> Any), O, O2](pipes: (F, O) => O2*)(evidence$2: Concurrent[F]): (F, O) => O2
Like apply but instead of providing a stream of worker streams, it runs each inner stream through
the supplied pipes.
the supplied pipes.
Supplied pipes are run concurrently with each other. Hence, the number of pipes determines concurrency.
Also, this guarantees that each pipe will view all
Also, this guarantees that each pipe will view all
O pulled from source stream, unlike broadcast.Resulting values are collected and returned in a single stream of
O2 values.- Value Params
- pipes
-
pipes that will concurrently process the work