appmaster

Type Members

1. class AppMaster extends ApplicationMaster

   

   AppMaster is the head of a streaming application.

   AppMaster is the head of a streaming application.

   It contains:

   1. ExecutorManager to manage all executors. 2. TaskManager to manage all tasks, 3. ClockService to track the global clock for this streaming application. 4. Scheduler to decide which a task should be scheduled to.

2. class ClockService extends Actor with Stash

   

   Maintains a global view of message timestamp in the application

3. class DagManager extends Actor with Stash

   

   Handles dag modification and other stuff related with DAG

   Handles dag modification and other stuff related with DAG

   DagManager maintains multiple version of DAGs. For each version, the DAG is immutable. For operations like modifying a processor, it will create a new version of DAG.

4. class JarScheduler extends AnyRef

   

   Different processors of the stream application can use different jars.

   Different processors of the stream application can use different jars. JarScheduler is the scheduler for different jars.

   For a DAG of multiple processors, each processor can have its own jar. Tasks of same jar is scheduled by TaskScheduler, and TaskSchedulers are scheduled by JarScheduler.

   In runtime, the implementation is delegated to actor JarSchedulerImpl

5. case class ProcessorSummary(id: ProcessorId, taskClass: String, parallelism: Int, description: String, taskConf: UserConfig, life: LifeTime, executors: List[ExecutorId], taskCount: Map[ExecutorId, TaskCount]) extends Product with Serializable

   

6. case class StreamAppMasterSummary(appType: String = "streaming", appId: Int, appName: String = null, actorPath: String = null, clock: TimeStamp = 0L, status: ApplicationStatus = ApplicationStatus.ACTIVE, startTime: TimeStamp = 0L, uptime: TimeStamp = 0L, user: String = null, homeDirectory: String = "", logFile: String = "", dag: Graph[ProcessorId, String] = null, executors: List[ExecutorBrief] = null, processors: Map[ProcessorId, ProcessorSummary] = ..., processorLevels: Map[ProcessorId, Int] = Map.empty[ProcessorId, Int], historyMetricsConfig: HistoryMetricsConfig = null) extends AppMasterSummary with Product with Serializable

   

   Stream application summary, used for REST API

7. case class TaskCount(count: Int) extends Product with Serializable

   

8. class TaskLocator extends AnyRef

   

   TaskLocator is used to decide which machine one task should run on.

   TaskLocator is used to decide which machine one task should run on.

   User can config org.apache.gearpump.streaming.Constants#GEARPUMP_STREAMING_LOCALITIES to decide to control which machine the task is running on.

9. class TaskRegistry extends AnyRef

   

   Tracks the registration of all tasks, when the application is booting up.

10. trait TaskScheduler extends AnyRef

    

    Schedules tasks to run for new allocated resources.

    Schedules tasks to run for new allocated resources. TaskScheduler only schedule tasks that share the same jar. For scheduling for multiple jars, see org.apache.gearpump.streaming.appmaster.JarScheduler.

11. class TaskSchedulerImpl extends TaskScheduler