io.smartdatalake.workflow.dataobject
KafkaTopicDataObject
Related Docs:
object KafkaTopicDataObject
| package dataobject
io.smartdatalake.workflow.dataobject
The name of the topic to read
Optional type the key column should be converted to. If none is given it will remain a bytearray / binary.
Optional type the value column should be converted to. If none is given it will remain a bytearray / binary.
An optional, minimal schema that this DataObject must have to pass schema validation on reading and writing.
Columns to be selected when reading the DataFrame. Available columns are key, value, topic, partition, offset, timestamp, timestampType. If key/valueType is AvroSchemaRegistry the key/value column are convert to a complex type according to the avro schema. To expand it select "value.*". Default is to select key and value.
definition of date partition column to extract formatted timestamp into column. This is used to list existing partition and is added as additional column on batch read.
Set to true if consecutive partitions should be combined as one range of offsets when batch reading from topic. This results in less tasks but can be a performance problem when reading many partitions. (default=false)
Set number of offsets per Spark task when batch reading from topic.
Options for the Kafka stream reader (see https://spark.apache.org/docs/latest/structured-streaming-kafka-integration.html). These options override connection.kafkaOptions.
Set to true if consecutive partitions should be combined as one range of offsets when batch reading from topic.
Set to true if consecutive partitions should be combined as one range of offsets when batch reading from topic. This results in less tasks but can be a performance problem when reading many partitions. (default=false)
Set number of offsets per Spark task when batch reading from topic.
Options for the Kafka stream reader (see https://spark.apache.org/docs/latest/structured-streaming-kafka-integration.html).
Options for the Kafka stream reader (see https://spark.apache.org/docs/latest/structured-streaming-kafka-integration.html). These options override connection.kafkaOptions.
definition of date partition column to extract formatted timestamp into column.
definition of date partition column to extract formatted timestamp into column. This is used to list existing partition and is added as additional column on batch read.
<invalid inheritdoc annotation>
<invalid inheritdoc annotation>
Optional type the key column should be converted to.
Optional type the key column should be converted to. If none is given it will remain a bytearray / binary.
An optional, minimal schema that this DataObject must have to pass schema validation on reading and writing.
An optional, minimal schema that this DataObject must have to pass schema validation on reading and writing.
Columns to be selected when reading the DataFrame.
Columns to be selected when reading the DataFrame. Available columns are key, value, topic, partition, offset, timestamp, timestampType. If key/valueType is AvroSchemaRegistry the key/value column are convert to a complex type according to the avro schema. To expand it select "value.*". Default is to select key and value.
The name of the topic to read
Optional type the value column should be converted to.
Optional type the value column should be converted to. If none is given it will remain a bytearray / binary.
DataObject of type KafkaTopic. Provides details to an action to read from Kafka Topics using either org.apache.spark.sql.DataFrameReader or org.apache.spark.sql.streaming.DataStreamReader
The name of the topic to read
Optional type the key column should be converted to. If none is given it will remain a bytearray / binary.
Optional type the value column should be converted to. If none is given it will remain a bytearray / binary.
An optional, minimal schema that this DataObject must have to pass schema validation on reading and writing.
Columns to be selected when reading the DataFrame. Available columns are key, value, topic, partition, offset, timestamp, timestampType. If key/valueType is AvroSchemaRegistry the key/value column are convert to a complex type according to the avro schema. To expand it select "value.*". Default is to select key and value.
definition of date partition column to extract formatted timestamp into column. This is used to list existing partition and is added as additional column on batch read.
Set to true if consecutive partitions should be combined as one range of offsets when batch reading from topic. This results in less tasks but can be a performance problem when reading many partitions. (default=false)
Set number of offsets per Spark task when batch reading from topic.
Options for the Kafka stream reader (see https://spark.apache.org/docs/latest/structured-streaming-kafka-integration.html). These options override connection.kafkaOptions.