com.coxautodata.waimak.dataflow
ParallelActionScheduler
Related Docs:
object ParallelActionScheduler
| package dataflow
com.coxautodata.waimak.dataflow
details of the execution pools: name, limits, running actions, thread pool
thread safe queue through which threads that have finished actions will communicate back to the scheduler
thread safe queue through which threads that have finished actions will communicate back to the scheduler
finds execution pools that have slots to run actions.
finds execution pools that have slots to run actions.
None if none of the pools are available for scheduling and Some(Set[AVAILABLE POOL NAME]) - all pools that are available for scheduling
Removes actions that are already running in the specified set pools.
Removes actions that are already running in the specified set pools.
pool names to which the from actions belong to
list of actions from poolNames that DataFlow knows have not been marked as executed and can be scheduled
list of actions that are not currently running
Checks if there are actions running at all, regardless of the execution pool.
Checks if there are actions running at all, regardless of the execution pool.
true if at least one action is running in any of the pools
Takes a value of type A and a msg to log, returning a and logging the message at the desired level
Takes a value of type A and a msg to log, returning a and logging the message at the desired level
a
Takes a value of type A and a function message from A to String, logs the value of invoking message(a) at the level described by the level parameter
Takes a value of type A and a function message from A to String, logs the value of invoking message(a) at the level described by the level parameter
a
logAndReturn(1, (num: Int) => s"number: $num", Info) // In the log we would see a log corresponding to "number 1"
details of the execution pools: name, limits, running actions, thread pool
Submits action into the specified execution pool.
Submits action into the specified execution pool.
pool into which to schedule the action
action to schedule
action labels that have data
object that allows access to the context of the flow and application
object that is used to signal start and end of the action execution
Executors must call it before exiting the execution of the flow to release resources.
Executors must call it before exiting the execution of the flow to release resources.
Locks and waits for at least one action to finish running, can return more than one action if they have finished and their results are available.
Locks and waits for at least one action to finish running, can return more than one action if they have finished and their results are available.
object that allows access to the context of the flow and application
object that is used to signal start and end of the action execution
Can run multiple actions in parallel with multiple execution pools support.
It was originally designed to benefit from Spark fair scheduler https://spark.apache.org/docs/latest/job-scheduling.html#fair-scheduler-pools Execution pool names must be the same as the Fair Scheduler pool names and number of parallel jobs within a pool is the number of Java threads.
Example: to configure 2 pools for Spark Fair Scheduler following xml must be passed to spark :
<?xml version="1.0"?> <allocations> <pool name="high"> <schedulingMode>FAIR</schedulingMode> <weight>1000</weight> <minShare>0</minShare> </pool> <pool name="medium"> <schedulingMode>FAIR</schedulingMode> <weight>25</weight> <minShare>0</minShare> </pool> </allocations>
following configuration options need to be specified:
in code pools parameter: Map( ("high" -> ExecutionPoolDesc("high", 10, Set.Empty, None)), ("medium" -> ExecutionPoolDesc("medium", 20, Set.Empty, None)) )
Created by Alexei Perelighin on 2018/07/10