cuttle

Type Members

class CancellationListener extends AnyRef

Allows to unsubscribe a cancel listener.
Allows to unsubscribe a cancel listener. It is sometimes useful for long running executions to dettach the cancel listener to not leak any memory.
sealed trait Completed extends AnyRef

Used a return type for the Job side effect instead of Unit.
Used a return type for the Job side effect instead of Unit.
We use Future[Completed] instead of Future[Unit] to avoid mistakes in user code because of value discarding.
case class DBLocation(host: String, port: Int) extends Product with Serializable

Configuration of JDBC endpoint.
Configuration of JDBC endpoint.
host
JDBC driver host
port
JDBC driver port
case class DatabaseConfig(locations: Seq[DBLocation], database: String, username: String, password: String) extends Product with Serializable

Configuration for the MySQL database used by Cuttle.
Configuration for the MySQL database used by Cuttle.
locations
sequence of JDBC endpoints
database
JDBC database
username
JDBC username
password
JDBC password
case class Execution[S <: Scheduling](id: String, job: Job[S], context: S.Context, streams: ExecutionStreams, platforms: Seq[ExecutionPlatform], projectName: String, projectVersion: String)(implicit executionContext: SideEffectThreadPool) extends Product with Serializable

Executions are created by the Scheduler.
Executions are created by the Scheduler.
id
The unique id for this execution. Guaranteed to be unique.
job
The Job for which this execution has been created.
context
The SchedulingContext for which this execution has been created.
streams
The execution streams are scoped stdout, stderr for the execution.
platforms
The available ExecutionPlatforms for this execution.
executionContext
The scoped scala.concurrent.ExecutionContext for this execution.
trait ExecutionPlatform extends AnyRef

An ExecutionPlatform provides controlled access to shared resources.
trait ExecutionStreams extends AnyRef

The scoped output streams for an Execution.
The scoped output streams for an Execution. Allows the execution to log its output.
class Executor[S <: Scheduling] extends MetricProvider[S]

An Executor is responsible to actually execute the SideEffect functions for the given Executions.
case class Job[S <: Scheduling](id: String, scheduling: S, name: String = "", description: String = "", tags: Set[Tag] = Set.empty[Tag])(effect: SideEffect[S]) extends Product with Serializable

The job SideEffect is the most important part as it represents the real job logic to execute.
The job SideEffect is the most important part as it represents the real job logic to execute. A job is defined for a given Scheduling, for example it can be a TimeSeries job. Jobs are also Workflow with a single vertice.
S
The kind of Scheduling used by this job.
id
The internal job id. It will be sued to track the job state in the database, so it must not change over time otherwise the job will be seen as a new one by the scheduler. That id, being technical, should only use valid characters such as [a-zA-Z0-9_-.]
scheduling
The scheduling configuration for the job. For example a TimeSeries job can be configured to be hourly or daily, etc.
name
The job name as displayed in the UI.
description
The job description as displayed in the UI.
tags
The job tags used to filter jobs in the UI.
effect
The job side effect, representing the real job execution.
trait Logger extends AnyRef

A logger used to output internal informations.
trait RetryStrategy extends AnyRef

The strategy to use to retry stuck executions.
The strategy to use to retry stuck executions.
When an Execution fails, the Executor keeps a track of this failure in a list of recently failed Executions.
If the Scheduler asks for the same Execution (same here is defined as an execution for the same Job and the same SchedulingContext) during the retryWindow defined by this strategy, it will be considered by the Executor as a stuck execution and delayed for the duration computed by the configured RetryStrategy.
trait Scheduler[S <: Scheduling] extends MetricProvider[S]

A scheduler interpret a Workflow and instanciate Executions for all defined Jobs.
A scheduler interpret a Workflow and instanciate Executions for all defined Jobs. For example, a typical cuttle Scheduler is the TimeSeries scheduler that executes the graph for each time partition.
S
The king of Scheduling managed by this Scheduler.
trait Scheduling extends AnyRef

Represent a type of scheduling.
Represent a type of scheduling. A typical cuttle scheduling is TimeSeries for example, that is a scheduling based on a calendar.
trait SchedulingContext extends AnyRef

A scheduling context is the input given to each Execution that will be created for a given Scheduling.
A scheduling context is the input given to each Execution that will be created for a given Scheduling. For example, for a TimeSeries scheduling the context contain the time partition for which the job is running.
type SideEffect[S <: Scheduling] = (Execution[S]) ⇒ Future[Completed]

The side effect function represents the real job execution.
The side effect function represents the real job execution. It returns a Future[Completed] to indicate the execution result (we use Completed here instead of Unit to avoid automatic value discarding, but Completed do not maintain additional state).
The cuttle Executor ensures that a scheduled side effect for a given SchedulingContext will be run a least once, but cannot guarantee that it will be run exactly once. That's why the side effect function must be idempotent, meaning that if executed for the same SchedulingContext it must produce the same result.
A failed future means a failed execution.
case class Tag(name: String, description: String = "") extends Product with Serializable

Allow to tag a job.
Allow to tag a job. Tags can be used in the UI/API to filter jobs and more easily retrieve them.
name
Tag name as displayed in the UI.
description
Description as displayed in the UI.
trait Workload[S <: Scheduling] extends AnyRef

Represent the workload of a Cuttle project, ie.
Represent the workload of a Cuttle project, ie. the list of jobs to be scheduled in some ways.
type XA = Transactor[IO]

Doobie transactor.
Doobie transactor. See https://github.com/tpolecat/doobie.

Value Members

object Auth

The cuttle API is private for any write operation while it is publicly open for any read only operation.
The cuttle API is private for any write operation while it is publicly open for any read only operation. It allows to make it easy to build tooling that monitor any running cuttle scheduler while restricting access to potentially dangerous operations.
The UI access itself requires authentication.
object Completed extends Completed with Product with Serializable

The object to use to successfully complete a job side effect.
The object to use to successfully complete a job side effect.
```
Future.successful(Completed)
```
object DatabaseConfig extends Serializable

Utilities for DatabaseConfig.
object ExecutionCancelled extends RuntimeException

Used to fail an execution Future when the execution has been cancelled.
object Job extends Product with Serializable

Companion object for Job.
object Metrics

Expose cuttle metrics via the prometheus protocol.
object Queries
object RetryStrategy

Built-in RetryStrategy
object SchedulingContext

Utilities for SchedulingContext
object Tag extends Serializable
object ThreadPools
package events
implicit val logger: Logger { def logMe(message: => String,level: String): Unit }

Default implicit logger that output everything to stdout
package platforms

Defines useful ExecutionPlatforms:
Defines useful ExecutionPlatforms:
- local to fork process locally in a managed way.
- http to make HTTP calls with rate limiting.
implicit def scopedExecutionContext(implicit execution: Execution[_]): ExecutionContext

Automatically provide a scala scala.concurrent.ExecutionContext for a given Execution.
Automatically provide a scala scala.concurrent.ExecutionContext for a given Execution. The threadpool will be chosen carefully by the Executor.
package utils

A set of basic utilities useful to write workflows.

package cuttle

Type Members

class CancellationListener extends AnyRef

sealed trait Completed extends AnyRef

case class DBLocation(host: String, port: Int) extends Product with Serializable

case class DatabaseConfig(locations: Seq[DBLocation], database: String, username: String, password: String) extends Product with Serializable

case class Execution[S <: Scheduling](id: String, job: Job[S], context: S.Context, streams: ExecutionStreams, platforms: Seq[ExecutionPlatform], projectName: String, projectVersion: String)(implicit executionContext: SideEffectThreadPool) extends Product with Serializable

trait ExecutionPlatform extends AnyRef

trait ExecutionStreams extends AnyRef

class Executor[S <: Scheduling] extends MetricProvider[S]

case class Job[S <: Scheduling](id: String, scheduling: S, name: String = "", description: String = "", tags: Set[Tag] = Set.empty[Tag])(effect: SideEffect[S]) extends Product with Serializable

trait Logger extends AnyRef

trait RetryStrategy extends AnyRef

trait Scheduler[S <: Scheduling] extends MetricProvider[S]

trait Scheduling extends AnyRef

trait SchedulingContext extends AnyRef

type SideEffect[S <: Scheduling] = (Execution[S]) ⇒ Future[Completed]

case class Tag(name: String, description: String = "") extends Product with Serializable

trait Workload[S <: Scheduling] extends AnyRef

type XA = Transactor[IO]

Value Members

object Auth

object Completed extends Completed with Product with Serializable

object DatabaseConfig extends Serializable

object ExecutionCancelled extends RuntimeException

object Job extends Product with Serializable

object Metrics

object Queries

object RetryStrategy

object SchedulingContext

object Tag extends Serializable

object ThreadPools

package events

implicit val logger: Logger { def logMe(message: => String,level: String): Unit }

package platforms

implicit def scopedExecutionContext(implicit execution: Execution[_]): ExecutionContext

package utils

Inherited from AnyRef

Inherited from Any

Ungrouped