abstract class Setl extends HasRegistry[Pipeline]
- Annotations
- @Evolving()
- Alphabetic
- By Inheritance
- Setl
- HasRegistry
- AnyRef
- Any
- Hide All
- Show All
- Public
- All
Instance Constructors
- new Setl(configLoader: ConfigLoader)
Abstract Value Members
- abstract val spark: SparkSession
Concrete Value Members
-
final
def
!=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
final
def
##(): Int
- Definition Classes
- AnyRef → Any
-
final
def
==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
final
def
asInstanceOf[T0]: T0
- Definition Classes
- Any
-
def
clearRegistry(): Unit
Clear the registry
Clear the registry
- Attributes
- protected
- Definition Classes
- HasRegistry
-
def
clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws( ... ) @native()
- val configLoader: ConfigLoader
-
final
def
eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
-
def
equals(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
def
finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws( classOf[java.lang.Throwable] )
-
final
def
getClass(): Class[_]
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
-
def
getConnector[CN <: Connector](connectorId: String): CN
Get a Connector.
Get a Connector. If the given config path hasn't been registered, then the connector will firstly be registered and then be returned.
- CN
type of the connector
- connectorId
id of connector (could be the config path)
- returns
the registered connector
-
def
getPipeline(uuid: UUID): Option[Pipeline]
Find a pipeline by its UUID
Find a pipeline by its UUID
- uuid
UUID of the target pipeline
-
def
getRegisteredItem(uuid: UUID): Option[Pipeline]
For a given UUID, return the corresponding registered item
-
def
getRegistry: ListMap[UUID, Pipeline]
Return the registry
Return the registry
- Definition Classes
- HasRegistry
-
def
getRegistryLength: Long
Return the number of items in the current registry
Return the number of items in the current registry
- Definition Classes
- HasRegistry
-
def
getSparkRepository[DT](repositoryId: String)(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[DT]): SparkRepository[DT]
Get a SparkRepository[DT].
Get a SparkRepository[DT]. If the given config path hasn't been registered, then the repository will firstly be registered and then be returned.
- DT
type of spark repository
- repositoryId
path to spark repository configuration
- returns
the added repository
-
def
hasRegisteredItem(uuid: UUID): Boolean
Check if the UUID exists in the registry
Check if the UUID exists in the registry
- uuid
an UUID
- returns
true if it already exists in the registry, false otherwise
- Definition Classes
- HasRegistry
-
def
hasRegisteredItem(item: Identifiable): Boolean
Check if the Identifiable exists in the registry
Check if the Identifiable exists in the registry
- item
an object that inherit io.github.setl.internal.Identifiable
- returns
true if it already exists in the registry, false otherwise
- Definition Classes
- HasRegistry
-
def
hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
-
final
def
isInstanceOf[T0]: Boolean
- Definition Classes
- Any
-
def
isRegistryEmpty: Boolean
Return true if the registry is empty, false otherwise
Return true if the registry is empty, false otherwise
- Definition Classes
- HasRegistry
-
def
lastRegisteredItem: Option[Pipeline]
Return the last registered item
Return the last registered item
- returns
if the registry is empty, None will be returned
- Definition Classes
- HasRegistry
-
final
def
ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
-
def
newPipeline(): Pipeline
Create a new pipeline.
Create a new pipeline. All the registered repositories and connectors will be passed into the delivery pool of the pipeline.
- returns
a newly instantiated pipeline object
-
final
def
notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
-
final
def
notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
-
def
registerNewItem(item: Pipeline): Unit
Register a new io.github.setl.internal.Identifiable in registry
Register a new io.github.setl.internal.Identifiable in registry
- item
an object that inherit io.github.setl.internal.Identifiable
- returns
true if the given item is registered, false otherwise
- Attributes
- protected
- Definition Classes
- HasRegistry
- Annotations
- @throws( ... )
-
def
registerNewItems(items: Iterable[Pipeline]): Unit
Register multiple items
Register multiple items
- items
an io.github.setl.internal.Identifiable object
- Attributes
- protected
- Definition Classes
- HasRegistry
-
def
resetConnector[CN <: Connector](connector: CN, deliveryId: String, connectorId: String)(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[CN]): Setl.this.type
Register a connector.
Register a connector.
If there this config path has been registered, it will be updated.
- CN
type of spark connector
- connector
a connector
- deliveryId
delivery ID
- returns
the current SETL context with the added repository
-
def
resetConnector[CN <: Connector](configPath: String, deliveryId: String, cls: Class[CN])(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[CN]): Setl.this.type
Register a connector.
Register a connector.
If there this config path has been registered, it will be updated.
- CN
type of spark connector
- configPath
path to connector configuration
- deliveryId
delivery ID
- cls
class of the Connector
- returns
the current SETL context with the added repository
-
def
resetSparkRepository[DT](config: String, consumer: Seq[Class[_ <: Factory[_]]] = Seq.empty, deliveryId: String = Deliverable.DEFAULT_ID, cacheData: Boolean = false)(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[DT]): Setl.this.type
Force register a spark repository for the given config path.
Force register a spark repository for the given config path. If there this config path has been registered, it will be updated
- DT
type of spark repository
- config
path to spark repository configuration
- consumer
Seq of consumer
- deliveryId
id of this delivery that will be used during the delivery matching
- cacheData
default false, if set to true, then the SparkRepository will cache the data after reading it
- returns
the current SETL context with the added repository
-
def
resetSparkRepository[DT](repository: SparkRepository[DT], consumer: Seq[Class[_ <: Factory[_]]], deliveryId: String, repositoryId: String)(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[DT]): Setl.this.type
Force register a spark repository with an object of SparkRepository and its id.
Force register a spark repository with an object of SparkRepository and its id. If a repository having the same ID was already registered, it will be overwritten by this one.
- DT
data type of the repository
- repository
an object of SparkRepository[T]
- consumer
consumer of this spark repository
- deliveryId
id of this delivery
- repositoryId
id to be used for the repository registration
- returns
the current SETL context with the added repository
-
def
setConnector[CN <: Connector](connector: CN, deliveryId: String, connectorId: String)(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[CN]): Setl.this.type
Register a connector.
Register a connector.
If there this config path has been registered, it will NOT be updated.
- CN
type of spark connector
- connector
a connector
- deliveryId
delivery ID
- connectorId
id of the Connector
- returns
the current SETL context with the added repository
-
def
setConnector[CN <: Connector](config: String, deliveryId: String, cls: Class[CN])(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[CN]): Setl.this.type
Register a connector.
Register a connector.
If there this config path has been registered, it will NOT be updated.
- CN
type of spark connector
- config
path to connector configuration
- deliveryId
delivery ID
- cls
class of the Connector
- returns
the current SETL context with the added repository
-
def
setConnector[CN <: Connector](config: String, cls: Class[CN])(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[CN]): Setl.this.type
Register a connector.
Register a connector. As each connector must have an delivery ID, by default the config path will be used.
If there this config path has been registered, it will NOT be updated.
- CN
type of connector
- config
path to connector configuration
- returns
the current SETL context with the added repository
-
def
setConnector(config: String, deliveryId: String): Setl.this.type
Register a connector.
Register a connector.
If there this config path has been registered, it will NOT be updated.
- config
path to connector configuration
- deliveryId
delivery ID
- returns
the current SETL context with the added connector
-
def
setConnector(config: String): Setl.this.type
Register a connector.
Register a connector. As each connector must have an delivery ID, by default the config path will be used.
If there this config path has been registered, it will NOT be updated.
- config
path to connector configuration
-
def
setSparkRepository[DT](repository: SparkRepository[DT], consumer: Seq[Class[_ <: Factory[_]]], deliveryId: String, repositoryId: String)(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[DT]): Setl.this.type
Register a spark repository with an object of SparkRepository and its id.
Register a spark repository with an object of SparkRepository and its id. If a repository having the same ID was already registered, it will NOT be overwritten by this one.
- DT
data type of the repository
- repository
an object of SparkRepository[T]
- consumer
consumer of this spark repository
- deliveryId
id of this delivery
- repositoryId
id to be used for the repository registration
- returns
the current SETL context with the added repository
-
def
setSparkRepository[DT](config: String, consumer: Seq[Class[_ <: Factory[_]]] = Seq.empty, deliveryId: String = Deliverable.DEFAULT_ID, cacheData: Boolean = false)(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[DT]): Setl.this.type
Register a spark repository for the given config path.
Register a spark repository for the given config path. If there this config path has been registered, it will NOT be updated
- DT
type of spark repository
- config
path to spark repository configuration
- consumer
Seq of consumer
- deliveryId
id of this delivery that will be used during the delivery matching
- cacheData
default false, if set to true, then the SparkRepository will cache the data after reading it
- returns
the current SETL context with the added repository
-
def
sparkSession: SparkSession
Return the current spark session
-
def
stop(): Unit
Stop the spark session
-
final
def
synchronized[T0](arg0: ⇒ T0): T0
- Definition Classes
- AnyRef
-
def
toString(): String
- Definition Classes
- AnyRef → Any
-
final
def
wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... )
-
final
def
wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... )
-
final
def
wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... ) @native()