DataControlJob

A simple flink job that transforms a data stream and a control stream into an output stream. This uses flink's CEP library to match sequences of data elements that fall between control elements that are alternately active and inactive. As an example, let on represent an active control, off represent an inactive control, and d to represent data elements. Then the following stream d1 d2 on d3 d4 d5 off d6 d7 would output one DataControlPeriod object with a start time of the timestamp of on, and end time of the timestamp of off and the elements d3, d4, d5 as the payload.

D: the data type
C: the control type
OUT: the output stream element type

Linear Supertypes

FlinkJob[DataControlPeriod[D], OUT], LazyLogging, AnyRef, Any

Instance Constructors

new DataControlJob()(implicit arg0: TypeInformation[D], arg1: TypeInformation[C], arg2: TypeInformation[OUT])

Abstract Value Members

abstract def transform(in: DataStream[DataControlPeriod[D]])(implicit config: FlinkConfig, env: SEE): DataStream[OUT]

Primary method to transform the source data stream into the output data stream.
Primary method to transform the source data stream into the output data stream. The output of this method is passed into sink(). This method must be overridden by subclasses.
in
input data stream created by source()
config
implicit flink job config
returns
output data stream

Definition Classes
FlinkJob

Concrete Value Members

final def !=(arg0: Any): Boolean

Definition Classes
AnyRef → Any
final def ##(): Int

Definition Classes
AnyRef → Any
final def ==(arg0: Any): Boolean

Definition Classes
AnyRef → Any
final def asInstanceOf[T0]: T0

Definition Classes
Any
def clone(): AnyRef

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( ... )
def control(implicit config: FlinkConfig, env: SEE): DataStream[C]

A source data stream for the control events.
A source data stream for the control events.
config
implicit flink config
returns
a data stream of control events.
def data(implicit config: FlinkConfig, env: SEE): DataStream[D]

A source data stream for the data events.
A source data stream for the data events.
config
implicit flink config
returns
a data stream of data events.
final def eq(arg0: AnyRef): Boolean

Definition Classes
AnyRef
def equals(arg0: Any): Boolean

Definition Classes
AnyRef → Any
def finalize(): Unit

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( classOf[java.lang.Throwable] )
def flow()(implicit config: FlinkConfig, env: SEE): DataStream[OUT]

A pipeline for transforming a single stream.
A pipeline for transforming a single stream. Passes the output of source() through transform() and the result of that into maybeSink(), which may pass it into sink() if we're not testing. Ultimately, returns the output data stream to facilitate testing.
config
implicit flink job config
returns
data output stream

Definition Classes
FlinkJob
final def getClass(): Class[_]

Definition Classes
AnyRef → Any
def hashCode(): Int

Definition Classes
AnyRef → Any
final def isInstanceOf[T0]: Boolean

Definition Classes
Any
lazy val logger: Logger

Attributes
protected
Definition Classes
LazyLogging
def maybeAssignTimestampsAndWatermarks(in: DataStream[DataControlPeriod[D]])(implicit config: FlinkConfig, env: SEE): Unit

Definition Classes
FlinkJob
def maybeSink(out: DataStream[OUT])(implicit config: FlinkConfig, env: SEE): Unit

The output stream will only be passed to sink() if FlinkConfig.mockEdges evaluates to false (ie, you're not testing).
The output stream will only be passed to sink() if FlinkConfig.mockEdges evaluates to false (ie, you're not testing).
out
the output data stream to pass into sink()
config
implicit flink job config

Definition Classes
FlinkJob
final def ne(arg0: AnyRef): Boolean

Definition Classes
AnyRef
final def notify(): Unit

Definition Classes
AnyRef
final def notifyAll(): Unit

Definition Classes
AnyRef
def pattern(implicit config: FlinkConfig): Pattern[DataOrControl[D, C], DataOrControl[D, C]]

Returns the pattern used by CEP to aggregate the control and data streams.
Returns the pattern used by CEP to aggregate the control and data streams. Should not need to be overridden.
returns
cep pattern
def run()(implicit config: FlinkConfig, env: SEE): Either[Iterator[OUT], Unit]

Definition Classes
FlinkJob
def sink(out: DataStream[OUT])(implicit config: FlinkConfig, env: SEE): Unit

Writes the transformed data stream to configured output sinks.
Writes the transformed data stream to configured output sinks. *
out
a transformed stream from transform()
config
implicit flink job config

Definition Classes
FlinkJob
def source()(implicit config: FlinkConfig, env: SEE): DataStream[DataControlPeriod[D]]

Generate a data stream of data control periods.
Generate a data stream of data control periods. This method does not generally need to be overridden in subclasses. It interleaves the data and control streams to produce a single stream of DataOrControl objects and then uses flink's CEP library to match sequences of control-on, data, and control-off events, which aggregates into a stream of DataControlPeriods.
At the moment, flink's CEP library has a bug in how its greedy operator works, which requires that we manually filter the control stream to remove multiple, sequential controls with the same isActive() value. So multiple on controls are replaced with the earliest on control and the same for off controls.
config
implicit flink configuration
returns
data stream of data control periods

Definition Classes
DataControlJob → FlinkJob
final def synchronized[T0](arg0: ⇒ T0): T0

Definition Classes
AnyRef
def toString(): String

Definition Classes
AnyRef → Any
final def wait(): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long, arg1: Int): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )

Related Docs: object DataControlJob | package flink

abstract class DataControlJob[D <: FlinkEvent, C <: FlinkEvent, OUT <: FlinkEvent] extends FlinkJob[DataControlPeriod[D], OUT]

Instance Constructors

new DataControlJob()(implicit arg0: TypeInformation[D], arg1: TypeInformation[C], arg2: TypeInformation[OUT])

Abstract Value Members

abstract def transform(in: DataStream[DataControlPeriod[D]])(implicit config: FlinkConfig, env: SEE): DataStream[OUT]

Concrete Value Members

final def !=(arg0: Any): Boolean

final def ##(): Int

final def ==(arg0: Any): Boolean

final def asInstanceOf[T0]: T0

def clone(): AnyRef

def control(implicit config: FlinkConfig, env: SEE): DataStream[C]

def data(implicit config: FlinkConfig, env: SEE): DataStream[D]

final def eq(arg0: AnyRef): Boolean

def equals(arg0: Any): Boolean

def finalize(): Unit

def flow()(implicit config: FlinkConfig, env: SEE): DataStream[OUT]

final def getClass(): Class[_]

def hashCode(): Int

final def isInstanceOf[T0]: Boolean

lazy val logger: Logger

def maybeAssignTimestampsAndWatermarks(in: DataStream[DataControlPeriod[D]])(implicit config: FlinkConfig, env: SEE): Unit

def maybeSink(out: DataStream[OUT])(implicit config: FlinkConfig, env: SEE): Unit

final def ne(arg0: AnyRef): Boolean

final def notify(): Unit

final def notifyAll(): Unit

def pattern(implicit config: FlinkConfig): Pattern[DataOrControl[D, C], DataOrControl[D, C]]

def run()(implicit config: FlinkConfig, env: SEE): Either[Iterator[OUT], Unit]

def sink(out: DataStream[OUT])(implicit config: FlinkConfig, env: SEE): Unit

def source()(implicit config: FlinkConfig, env: SEE): DataStream[DataControlPeriod[D]]

final def synchronized[T0](arg0: ⇒ T0): T0

def toString(): String

final def wait(): Unit

final def wait(arg0: Long, arg1: Int): Unit

final def wait(arg0: Long): Unit

Inherited from FlinkJob[DataControlPeriod[D], OUT]

Inherited from LazyLogging

Inherited from AnyRef

Inherited from Any

Ungrouped