Packages

c

org.apache.spark.sql.kafka010

KafkaContinuousInputPartition

case class KafkaContinuousInputPartition(topicPartition: TopicPartition, startOffset: Long, kafkaParams: Map[String, AnyRef], pollTimeoutMs: Long, failOnDataLoss: Boolean, includeHeaders: Boolean) extends InputPartition with Product with Serializable

An input partition for continuous Kafka processing. This will be serialized and transformed into a full reader on executors.

topicPartition

The (topic, partition) pair this task is responsible for.

startOffset

The offset to start reading from within the partition.

kafkaParams

Kafka consumer params to use.

pollTimeoutMs

The timeout for Kafka consumer polling.

failOnDataLoss

Flag indicating whether data reader should fail if some offsets are skipped.

includeHeaders

Flag indicating whether to include Kafka records' headers.

Linear Supertypes
Product, Equals, InputPartition, Serializable, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. KafkaContinuousInputPartition
  2. Product
  3. Equals
  4. InputPartition
  5. Serializable
  6. AnyRef
  7. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Instance Constructors

  1. new KafkaContinuousInputPartition(topicPartition: TopicPartition, startOffset: Long, kafkaParams: Map[String, AnyRef], pollTimeoutMs: Long, failOnDataLoss: Boolean, includeHeaders: Boolean)

    topicPartition

    The (topic, partition) pair this task is responsible for.

    startOffset

    The offset to start reading from within the partition.

    kafkaParams

    Kafka consumer params to use.

    pollTimeoutMs

    The timeout for Kafka consumer polling.

    failOnDataLoss

    Flag indicating whether data reader should fail if some offsets are skipped.

    includeHeaders

    Flag indicating whether to include Kafka records' headers.

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##: Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.CloneNotSupportedException]) @IntrinsicCandidate() @native()
  6. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  7. val failOnDataLoss: Boolean
  8. final def getClass(): Class[_ <: AnyRef]
    Definition Classes
    AnyRef → Any
    Annotations
    @IntrinsicCandidate() @native()
  9. val includeHeaders: Boolean
  10. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  11. val kafkaParams: Map[String, AnyRef]
  12. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  13. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @IntrinsicCandidate() @native()
  14. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @IntrinsicCandidate() @native()
  15. val pollTimeoutMs: Long
  16. def preferredLocations(): Array[String]
    Definition Classes
    InputPartition
  17. def productElementNames: Iterator[String]
    Definition Classes
    Product
  18. val startOffset: Long
  19. final def synchronized[T0](arg0: => T0): T0
    Definition Classes
    AnyRef
  20. val topicPartition: TopicPartition
  21. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  22. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException]) @native()
  23. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])

Deprecated Value Members

  1. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.Throwable]) @Deprecated
    Deprecated

    (Since version 9)

Inherited from Product

Inherited from Equals

Inherited from InputPartition

Inherited from Serializable

Inherited from AnyRef

Inherited from Any

Ungrouped