IgnitePipe
Related Docs:
object IgnitePipe
| package scala
Execute the chain and return the computed values.
Transform each value using the function f and flatten the result.
Transform each value using the function f and flatten the result.
Note: This is not a monadic composition.
Flatten step is performed on the client. If you have a chain of flatMaps, all functions in the chain are composed and flattening is performed once on the client.
To manually split the flatMap chain, use .fork. Forking is useful when dealing with long, lazy chains, or when adding a barrier is desired.
Manually add a fork in the execution chain.
Manually add a fork in the execution chain. This creates a barrier, which means the subsequent transforms are planned on a fresh Ignite closure.
Transform each element using the function f.
Transform each element using the function f.
This is executed on the cluster nodes. Chained map transforms are composed and executed once on the cluster nodes. Use .fork to manually split the chain if tuning is required.
Prepare a Reduction based on the provided Semigroup.
Prepare a Reduction based on the provided Semigroup.
Note that results can arrived from cluster nodes in any order, so the operation has to be associative and commutative.
Merge two pipes of the same type
Filter elements using the function f.
Filter elements using the function f.
Implemented as a flatMap executed on the client.
Provides composable distributed closures that can run on Apache Ignite.
Allows chaining functions to be executed on the cluster. Reduction is done on the client. Note that pipe operations like flattening, filtering are also performed on the client after gathering results from the nodes.
Best practice is to push computations to the cluster as much as possible and flatten, filter on the client only if the scatter-gather overhead is acceptable and results can fit on the client.