processors

case class FormatConverterContext[SourceConfig <: FormatAwareDataSourceConfiguration, SinkConfig <: FormatAwareDataSinkConfiguration](input: SourceConfig, output: SinkConfig) extends Product with Serializable

Context class for the FormatConverter
Context class for the FormatConverter
input
data source
output
data sink
abstract class SqlProcessor extends SparkApp[SqlProcessorContext[_ <: FormatAwareDataSourceConfiguration, _ <: FormatAwareDataSinkConfiguration], DataFrame]

The SqlProcessor is a base class that can support multiple implementation, mainly designed to support registering custom SQL functions (UDFs) to make them available while running the queries.
case class SqlProcessorContext[SourceConfig <: FormatAwareDataSourceConfiguration, SinkConfig <: FormatAwareDataSinkConfiguration](inputTables: Map[String, SourceConfig], inputVariables: Map[String, String], outputConfig: SinkConfig, sql: String) extends Product with Serializable

object FormatConverter extends SparkApp[FormatConverterContext[_ <: FormatAwareDataSourceConfiguration, _ <: FormatAwareDataSinkConfiguration], DataFrame]

Load a file into a DataFrame and save it as a file in the specified path.
Load a file into a DataFrame and save it as a file in the specified path.
- For the XML converter see more details here: https://github.com/databricks/spark-xml
- For the CSV converter see more details here: https://spark.apache.org/docs/2.1.1/api/java/org/apache/spark/sql/DataFrameReader.html#csv(java.lang.String...)
- For the JSON converter see more details here: https://spark.apache.org/docs/2.1.1/api/java/org/apache/spark/sql/DataFrameReader.html#json(java.lang.String...)
- For the AVRO converter see more details here: https://github.com/databricks/spark-avro
object FormatConverterContext extends Configurator[FormatConverterContext[_ <: FormatAwareDataSourceConfiguration, _ <: FormatAwareDataSinkConfiguration]] with Serializable
object SimpleSqlProcessor extends SqlProcessor

This is the most basic implementation of hte SQL processor and it registers no custom functions
object SqlProcessorContext extends Configurator[SqlProcessorContext[_ <: FormatAwareDataSourceConfiguration, _ <: FormatAwareDataSinkConfiguration]] with Logging with Serializable
package info