Packages

c

org.apache.spark.sql.kafka010

KafkaContinuousPartitionReader

class KafkaContinuousPartitionReader extends ContinuousPartitionReader[InternalRow]

A per-task data reader for continuous Kafka processing.

Linear Supertypes
ContinuousPartitionReader[InternalRow], PartitionReader[InternalRow], Closeable, AutoCloseable, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. KafkaContinuousPartitionReader
  2. ContinuousPartitionReader
  3. PartitionReader
  4. Closeable
  5. AutoCloseable
  6. AnyRef
  7. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Instance Constructors

  1. new KafkaContinuousPartitionReader(topicPartition: TopicPartition, startOffset: Long, kafkaParams: Map[String, AnyRef], pollTimeoutMs: Long, failOnDataLoss: Boolean, includeHeaders: Boolean)

    topicPartition

    The (topic, partition) pair this data reader is responsible for.

    startOffset

    The offset to start reading from within the partition.

    kafkaParams

    Kafka consumer params to use.

    pollTimeoutMs

    The timeout for Kafka consumer polling.

    failOnDataLoss

    Flag indicating whether data reader should fail if some offsets are skipped.

Value Members

  1. def close(): Unit
    Definition Classes
    KafkaContinuousPartitionReader → Closeable → AutoCloseable
  2. def currentMetricsValues(): Array[CustomTaskMetric]
    Definition Classes
    PartitionReader
  3. def get(): UnsafeRow
    Definition Classes
    KafkaContinuousPartitionReader → PartitionReader
  4. def getOffset(): KafkaSourcePartitionOffset
    Definition Classes
    KafkaContinuousPartitionReader → ContinuousPartitionReader
  5. def next(): Boolean
    Definition Classes
    KafkaContinuousPartitionReader → PartitionReader