Packages

o

org.apache.spark.sql.kafka010

KafkaMicroBatchStream

object KafkaMicroBatchStream extends Logging

Linear Supertypes
Logging, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. KafkaMicroBatchStream
  2. Logging
  3. AnyRef
  4. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Value Members

  1. def metrics(latestConsumedOffset: Optional[Offset], latestAvailablePartitionOffsets: PartitionOffsetMap): Map[String, String]

    Compute the difference of offset per partition between latestAvailablePartitionOffsets and partition offsets in the latestConsumedOffset.

    Compute the difference of offset per partition between latestAvailablePartitionOffsets and partition offsets in the latestConsumedOffset. Report min/max/avg offsets behind the latest for all the partitions in the Kafka stream.

    Because of rate limit, latest consumed offset per partition can be smaller than the latest available offset per partition.

    latestConsumedOffset

    latest consumed offset

    latestAvailablePartitionOffsets

    latest available offset per partition

    returns

    the generated metrics map