internal
Connectors that inherit CanCreate should be able to create a table in a database or a file/folder in a file system
Connectors that inherit CanDelete should be able to delete records for a given query string
Connectors that inherit CanDrop should be able to drop the entire data table
Connectors that inherit CanPartition should be able to partition the output by the given columns on the file system
Connectors that inherit CanUpdate should be able to update the data store with a new data frame and a given matching columns.
Connectors that inherit CanVacuum should be able to recursively delete files and directories in the table that are not needed by the table for maintaining older versions up to the given retention threshold
Connectors that inherit CanWait should be able to wait for the execution to stop
HasBenchmark should be used for object having an aggregated benchmark.
HasBenchmark should be used for object having an aggregated benchmark. Typically a Pipeline or a Stage
HasUUIDRegistry provide a UUID registry and methods to check if an io.github.setl.internal.Identifiable object already exists in its registry
HasUUIDRegistry provide a UUID registry and methods to check if an io.github.setl.internal.Identifiable object already exists in its registry
HasType should be used on classed having a payload
HasType should be used on classed having a payload
Identifiable generates an UUID for any object that implement the trait
Identifiable generates an UUID for any object that implement the trait
Indicate that users can activate or deactivate the write of the class
SchemaConverter will rename the column of a dataset/dataframe according to the given case class T.
SchemaConverter will rename the column of a dataset/dataframe according to the given case class T.
import io.github.setl.annotations.colName case class MyObject(@colName("col1") column1: String, column2: String) val ds: Dataset[MyObject] = Seq(MyObject("a", "A"), MyObject("b", "B")).toDS() // +-------+-------+ // |column1|column2| // +-------+-------+ // | a| A| // | b| B| // +-------+-------+ val df = SchemaConverter.toDF(ds) // +----+-------+ // |col1|column2| // +----+-------+ // | a| A| // | b| B| // +----+-------+ val ds2 = SchemaConverter.fromDF[MyObject](df) // +-------+-------+ // |column1|column2| // +-------+-------+ // | a| A| // | b| B| // +-------+-------+
StructAnalyser will analyse the schema for a given case class.
StructAnalyser will analyse the schema for a given case class. It will register the information about