FilterByControlJob

A simple flink job that interleaves a data stream and a control stream and creates a source of data records that occur during active control periods.

As an example, let on represent an active control, off represent an inactive control, and d to represent data elements. Then the following stream:

d1 d2 on d3 d4 d5 off d6 d7

would output d3 d4 d5.

D: the data type
C: the control type
OUT: the output stream element type

Linear Supertypes

FlinkJob[D, OUT], LazyLogging, AnyRef, Any

Instance Constructors

new FilterByControlJob()(implicit arg0: TypeInformation[D], arg1: TypeInformation[C], arg2: TypeInformation[OUT])

Abstract Value Members

abstract def transform(in: DataStream[D])(implicit config: FlinkConfig, env: SEE): DataStream[OUT]

Primary method to transform the source data stream into the output data stream.
Primary method to transform the source data stream into the output data stream. The output of this method is passed into sink(). This method must be overridden by subclasses.
in
input data stream created by source()
config
implicit flink job config
returns
output data stream

Definition Classes
FlinkJob

Concrete Value Members

final def !=(arg0: Any): Boolean

Definition Classes
AnyRef → Any
final def ##(): Int

Definition Classes
AnyRef → Any
final def ==(arg0: Any): Boolean

Definition Classes
AnyRef → Any
final def asInstanceOf[T0]: T0

Definition Classes
Any
def clone(): AnyRef

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( ... )
def control(implicit config: FlinkConfig, env: SEE): DataStream[C]

A source data stream for the control events.
A source data stream for the control events.
config
implicit flink config
returns
a data stream of control events.
def data(implicit config: FlinkConfig, env: SEE): DataStream[D]

A source data stream for the data events.
A source data stream for the data events.
config
implicit flink config
returns
a data stream of data events.
final def eq(arg0: AnyRef): Boolean

Definition Classes
AnyRef
def equals(arg0: Any): Boolean

Definition Classes
AnyRef → Any
def finalize(): Unit

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( classOf[java.lang.Throwable] )
def flow()(implicit config: FlinkConfig, env: SEE): DataStream[OUT]

A pipeline for transforming a single stream.
A pipeline for transforming a single stream. Passes the output of source() through transform() and the result of that into maybeSink(), which may pass it into sink() if we're not testing. Ultimately, returns the output data stream to facilitate testing.
config
implicit flink job config
returns
data output stream

Definition Classes
FlinkJob
final def getClass(): Class[_]

Definition Classes
AnyRef → Any
def hashCode(): Int

Definition Classes
AnyRef → Any
final def isInstanceOf[T0]: Boolean

Definition Classes
Any
lazy val logger: Logger

Attributes
protected
Definition Classes
LazyLogging
def maybeAssignTimestampsAndWatermarks(in: DataStream[D])(implicit config: FlinkConfig, env: SEE): Unit

Definition Classes
FlinkJob
def maybeSink(out: DataStream[OUT])(implicit config: FlinkConfig, env: SEE): Unit

The output stream will only be passed to sink() if FlinkConfig.mockEdges evaluates to false (ie, you're not testing).
The output stream will only be passed to sink() if FlinkConfig.mockEdges evaluates to false (ie, you're not testing).
out
the output data stream to pass into sink()
config
implicit flink job config

Definition Classes
FlinkJob
final def ne(arg0: AnyRef): Boolean

Definition Classes
AnyRef
final def notify(): Unit

Definition Classes
AnyRef
final def notifyAll(): Unit

Definition Classes
AnyRef
def run()(implicit config: FlinkConfig, env: SEE): Either[Iterator[OUT], Unit]

Definition Classes
FlinkJob
def sink(out: DataStream[OUT])(implicit config: FlinkConfig, env: SEE): Unit

Writes the transformed data stream to configured output sinks.
Writes the transformed data stream to configured output sinks. *
out
a transformed stream from transform()
config
implicit flink job config

Definition Classes
FlinkJob
def source()(implicit config: FlinkConfig, env: SEE): DataStream[D]

Generate a stream of data records filtered by the control stream.
Generate a stream of data records filtered by the control stream. This method does not generally need to be overridden in subclasses. It interleaves the data and control streams to produce a single stream of DataOrControl objects and then uses a flat map with state to determine when to emit the data records. It remembers the last control time and state and updates it when the state changes. *
config
implicit flink config
returns
data stream of data records

Definition Classes
FilterByControlJob → FlinkJob
final def synchronized[T0](arg0: ⇒ T0): T0

Definition Classes
AnyRef
def toString(): String

Definition Classes
AnyRef → Any
final def wait(): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long, arg1: Int): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )

Related Doc: package flink

abstract class FilterByControlJob[D <: FlinkEvent, C <: FlinkEvent, OUT <: FlinkEvent] extends FlinkJob[D, OUT]

Instance Constructors

new FilterByControlJob()(implicit arg0: TypeInformation[D], arg1: TypeInformation[C], arg2: TypeInformation[OUT])

Abstract Value Members

abstract def transform(in: DataStream[D])(implicit config: FlinkConfig, env: SEE): DataStream[OUT]

Concrete Value Members

final def !=(arg0: Any): Boolean

final def ##(): Int

final def ==(arg0: Any): Boolean

final def asInstanceOf[T0]: T0

def clone(): AnyRef

def control(implicit config: FlinkConfig, env: SEE): DataStream[C]

def data(implicit config: FlinkConfig, env: SEE): DataStream[D]

final def eq(arg0: AnyRef): Boolean

def equals(arg0: Any): Boolean

def finalize(): Unit

def flow()(implicit config: FlinkConfig, env: SEE): DataStream[OUT]

final def getClass(): Class[_]

def hashCode(): Int

final def isInstanceOf[T0]: Boolean

lazy val logger: Logger

def maybeAssignTimestampsAndWatermarks(in: DataStream[D])(implicit config: FlinkConfig, env: SEE): Unit

def maybeSink(out: DataStream[OUT])(implicit config: FlinkConfig, env: SEE): Unit

final def ne(arg0: AnyRef): Boolean

final def notify(): Unit

final def notifyAll(): Unit

def run()(implicit config: FlinkConfig, env: SEE): Either[Iterator[OUT], Unit]

def sink(out: DataStream[OUT])(implicit config: FlinkConfig, env: SEE): Unit

def source()(implicit config: FlinkConfig, env: SEE): DataStream[D]

final def synchronized[T0](arg0: ⇒ T0): T0

def toString(): String

final def wait(): Unit

final def wait(arg0: Long, arg1: Int): Unit

final def wait(arg0: Long): Unit

Inherited from FlinkJob[D, OUT]

Inherited from LazyLogging

Inherited from AnyRef

Inherited from Any

Ungrouped