EnrichmentAsyncFunction

An abstract asynchronous function to enrich a data stream with non-stream data. This class relies on guava's CacheBuilder to load and cache the enrichment data. The default cache loader assumes the cache key is a uri of a json api endpoint which it loads asynchronously and converts to the cache value type (CV). You can provide your own cache loader to load enrichment data in some other way. If you use the default loader, note you must implicitly provide a circe EntityDecoder to convert the json api result body to the cache value type.

The behavior of the function can be parameterized with flink config values for

$configPrefix.num.threads (size of thread pool, Int)
$configPrefix.cache.concurrency.level (Int)
$configPrefix.cache.max.size (max number of records in cache, Int)
$configPrefix.cache.expire.after (Duration)

The cache always uses weak keys, allowing for aggressive garbage collection of unused values.

Subclasses must implement the getCacheKey() and enrichEvent() methods.

IN: the input stream element type
OUT: the enriched stream output element type
CV: the cache value type

Linear Supertypes

LazyLogging, AsyncFunction[IN, OUT], Function, Serializable, AnyRef, Any

Instance Constructors

new EnrichmentAsyncFunction(configPrefix: String, cacheLoaderOpt: Option[CacheLoader[String, Option[CV]]] = None, preloaded: Map[String, CV] = Map.empty[String, CV])(implicit config: FlinkConfig, decoder: Decoder[CV])

configPrefix
for extracting configuration information
cacheLoaderOpt
an optional CacheLoader for loading the enrichment data
config
implicit flink config
decoder
implicit entity decoder for converting the body of the api call to the cache value type

Abstract Value Members

abstract def enrichEvent(in: IN, data: Option[CV]): Seq[OUT]

Construct a sequence of zero or more enriched output events using the input event and the api results.
Construct a sequence of zero or more enriched output events using the input event and the api results.
in
input event
data
some api results (or none if api call failed)
abstract def getCacheKey(in: IN): String

Generate the cache key from the input event.
Generate the cache key from the input event. For the default cache loader implementation, this should be a json api endpoint uri. If you provide your own cache loader implementation, this should be whatever is appropriate, however, it must be a String.
in
the input event

Concrete Value Members

final def !=(arg0: Any): Boolean

Definition Classes
AnyRef → Any
final def ##(): Int

Definition Classes
AnyRef → Any
final def ==(arg0: Any): Boolean

Definition Classes
AnyRef → Any
lazy val api: Resource[IO, Client[IO]]
final def asInstanceOf[T0]: T0

Definition Classes
Any
def asyncInvoke(in: IN, collector: ResultFuture[OUT]): Unit

Definition Classes
EnrichmentAsyncFunction → AsyncFunction
def asyncInvokeF(in: IN): Future[Try[Seq[OUT]]]

A helper method to enable testing of asyncInvoke() without needing to construct a flink ResultFuture collector.
A helper method to enable testing of asyncInvoke() without needing to construct a flink ResultFuture collector.
in
the input event
lazy val cache: LoadingCache[String, Option[CV]]
def clone(): AnyRef

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( ... )
implicit lazy val cs: ContextShift[IO]
lazy val defaultCacheLoader: CacheLoader[String, Option[CV]]

The default cache loader implementation.
The default cache loader implementation. This uses a blaze client to make an api call and converts the result to the cache value type (CV). This loader will be used unless one is passed into the class constructor (usually just done for testing).
implicit lazy val ec: ExecutionContext

A thread pool execution context for making asynchronous api calls.
implicit lazy val entityDecoder: EntityDecoder[IO, CV]
final def eq(arg0: AnyRef): Boolean

Definition Classes
AnyRef
def equals(arg0: Any): Boolean

Definition Classes
AnyRef → Any
def finalize(): Unit

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( classOf[java.lang.Throwable] )
final def getClass(): Class[_]

Definition Classes
AnyRef → Any
def getConfigPrefix: String

Getter for $configPrefix value
def hashCode(): Int

Definition Classes
AnyRef → Any
final def isInstanceOf[T0]: Boolean

Definition Classes
Any
lazy val logger: Logger

Attributes
protected
Definition Classes
LazyLogging
final def ne(arg0: AnyRef): Boolean

Definition Classes
AnyRef
final def notify(): Unit

Definition Classes
AnyRef
final def notifyAll(): Unit

Definition Classes
AnyRef
final def synchronized[T0](arg0: ⇒ T0): T0

Definition Classes
AnyRef
def timeout(input: IN, resultFuture: ResultFuture[OUT]): Unit

Definition Classes
AsyncFunction
implicit lazy val timer: Timer[IO]
def toString(): String

Definition Classes
AnyRef → Any
final def wait(): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long, arg1: Int): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )

Related Doc: package operator

abstract class EnrichmentAsyncFunction[IN, OUT, CV <: AnyRef] extends AsyncFunction[IN, OUT] with LazyLogging

Instance Constructors

new EnrichmentAsyncFunction(configPrefix: String, cacheLoaderOpt: Option[CacheLoader[String, Option[CV]]] = None, preloaded: Map[String, CV] = Map.empty[String, CV])(implicit config: FlinkConfig, decoder: Decoder[CV])

Abstract Value Members

abstract def enrichEvent(in: IN, data: Option[CV]): Seq[OUT]

abstract def getCacheKey(in: IN): String

Concrete Value Members

final def !=(arg0: Any): Boolean

final def ##(): Int

final def ==(arg0: Any): Boolean

lazy val api: Resource[IO, Client[IO]]

final def asInstanceOf[T0]: T0

def asyncInvoke(in: IN, collector: ResultFuture[OUT]): Unit

def asyncInvokeF(in: IN): Future[Try[Seq[OUT]]]

lazy val cache: LoadingCache[String, Option[CV]]

def clone(): AnyRef

implicit lazy val cs: ContextShift[IO]

lazy val defaultCacheLoader: CacheLoader[String, Option[CV]]

implicit lazy val ec: ExecutionContext

implicit lazy val entityDecoder: EntityDecoder[IO, CV]

final def eq(arg0: AnyRef): Boolean

def equals(arg0: Any): Boolean

def finalize(): Unit

final def getClass(): Class[_]

def getConfigPrefix: String

def hashCode(): Int

final def isInstanceOf[T0]: Boolean

lazy val logger: Logger

final def ne(arg0: AnyRef): Boolean

final def notify(): Unit

final def notifyAll(): Unit

final def synchronized[T0](arg0: ⇒ T0): T0

def timeout(input: IN, resultFuture: ResultFuture[OUT]): Unit

implicit lazy val timer: Timer[IO]

def toString(): String

final def wait(): Unit

final def wait(arg0: Long, arg1: Int): Unit

final def wait(arg0: Long): Unit

Inherited from LazyLogging

Inherited from AsyncFunction[IN, OUT]

Inherited from Function

Inherited from Serializable

Inherited from AnyRef

Inherited from Any

Ungrouped