Packages

c

org.apache.spark.sql.kafka010

KafkaContinuousInputPartition

case class KafkaContinuousInputPartition(topicPartition: TopicPartition, startOffset: Long, kafkaParams: Map[String, AnyRef], pollTimeoutMs: Long, failOnDataLoss: Boolean, includeHeaders: Boolean) extends InputPartition with Product with Serializable

An input partition for continuous Kafka processing. This will be serialized and transformed into a full reader on executors.

topicPartition

The (topic, partition) pair this task is responsible for.

startOffset

The offset to start reading from within the partition.

kafkaParams

Kafka consumer params to use.

pollTimeoutMs

The timeout for Kafka consumer polling.

failOnDataLoss

Flag indicating whether data reader should fail if some offsets are skipped.

includeHeaders

Flag indicating whether to include Kafka records' headers.

Linear Supertypes
Product, Equals, InputPartition, Serializable, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. KafkaContinuousInputPartition
  2. Product
  3. Equals
  4. InputPartition
  5. Serializable
  6. AnyRef
  7. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Instance Constructors

  1. new KafkaContinuousInputPartition(topicPartition: TopicPartition, startOffset: Long, kafkaParams: Map[String, AnyRef], pollTimeoutMs: Long, failOnDataLoss: Boolean, includeHeaders: Boolean)

    topicPartition

    The (topic, partition) pair this task is responsible for.

    startOffset

    The offset to start reading from within the partition.

    kafkaParams

    Kafka consumer params to use.

    pollTimeoutMs

    The timeout for Kafka consumer polling.

    failOnDataLoss

    Flag indicating whether data reader should fail if some offsets are skipped.

    includeHeaders

    Flag indicating whether to include Kafka records' headers.

Value Members

  1. val failOnDataLoss: Boolean
  2. val includeHeaders: Boolean
  3. val kafkaParams: Map[String, AnyRef]
  4. val pollTimeoutMs: Long
  5. def preferredLocations(): Array[String]
    Definition Classes
    InputPartition
  6. def productElementNames: Iterator[String]
    Definition Classes
    Product
  7. val startOffset: Long
  8. val topicPartition: TopicPartition