Packages

c

org.apache.spark.sql.kafka010

KafkaContinuousInputPartition

case class KafkaContinuousInputPartition(topicPartition: TopicPartition, startOffset: Long, kafkaParams: Map[String, AnyRef], pollTimeoutMs: Long, failOnDataLoss: Boolean, includeHeaders: Boolean) extends InputPartition with Product with Serializable

An input partition for continuous Kafka processing. This will be serialized and transformed into a full reader on executors.

topicPartition

The (topic, partition) pair this task is responsible for.

startOffset

The offset to start reading from within the partition.

kafkaParams

Kafka consumer params to use.

pollTimeoutMs

The timeout for Kafka consumer polling.

failOnDataLoss

Flag indicating whether data reader should fail if some offsets are skipped.

includeHeaders

Flag indicating whether to include Kafka records' headers.

Linear Supertypes
Serializable, Product, Equals, InputPartition, Serializable, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. KafkaContinuousInputPartition
  2. Serializable
  3. Product
  4. Equals
  5. InputPartition
  6. Serializable
  7. AnyRef
  8. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new KafkaContinuousInputPartition(topicPartition: TopicPartition, startOffset: Long, kafkaParams: Map[String, AnyRef], pollTimeoutMs: Long, failOnDataLoss: Boolean, includeHeaders: Boolean)

    topicPartition

    The (topic, partition) pair this task is responsible for.

    startOffset

    The offset to start reading from within the partition.

    kafkaParams

    Kafka consumer params to use.

    pollTimeoutMs

    The timeout for Kafka consumer polling.

    failOnDataLoss

    Flag indicating whether data reader should fail if some offsets are skipped.

    includeHeaders

    Flag indicating whether to include Kafka records' headers.

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  6. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  7. val failOnDataLoss: Boolean
  8. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  9. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  10. val includeHeaders: Boolean
  11. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  12. val kafkaParams: Map[String, AnyRef]
  13. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  14. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  15. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  16. val pollTimeoutMs: Long
  17. def preferredLocations(): Array[String]
    Definition Classes
    InputPartition
  18. val startOffset: Long
  19. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  20. val topicPartition: TopicPartition
  21. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  22. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  23. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()

Inherited from Serializable

Inherited from Product

Inherited from Equals

Inherited from InputPartition

Inherited from Serializable

Inherited from AnyRef

Inherited from Any

Ungrouped