Trait

org.apache.spark.sql.execution.streaming

ProgressReporter

Related Doc: package streaming

Permalink

trait ProgressReporter extends Logging

Responsible for continually reporting statistics about the amount of data processed as well as latency for a streaming query. This trait is designed to be mixed into the StreamExecution, who is responsible for calling startTrigger and finishTrigger at the appropriate times. Additionally, the status can updated with updateStatusMessage to allow reporting on the streams current state (i.e. "Fetching more data").

Linear Supertypes
Logging, AnyRef, Any
Known Subclasses
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. ProgressReporter
  2. Logging
  3. AnyRef
  4. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Type Members

  1. case class ExecutionStats(inputRows: Map[Source, Long], stateOperators: Seq[StateOperatorProgress], eventTimeStats: Map[String, String]) extends Product with Serializable

    Permalink

Abstract Value Members

  1. abstract def availableOffsets: StreamProgress

    Permalink
    Attributes
    protected
  2. abstract def committedOffsets: StreamProgress

    Permalink
    Attributes
    protected
  3. abstract def currentBatchId: Long

    Permalink
    Attributes
    protected
  4. abstract def id: UUID

    Permalink
    Attributes
    protected
  5. abstract def lastExecution: QueryExecution

    Permalink
    Attributes
    protected
  6. abstract def logicalPlan: LogicalPlan

    Permalink
    Attributes
    protected
  7. abstract def name: String

    Permalink
    Attributes
    protected
  8. abstract def newData: Map[Source, DataFrame]

    Permalink
    Attributes
    protected
  9. abstract def offsetSeqMetadata: OffsetSeqMetadata

    Permalink
    Attributes
    protected
  10. abstract def postEvent(event: Event): Unit

    Permalink
    Attributes
    protected
  11. abstract def runId: UUID

    Permalink
    Attributes
    protected
  12. abstract def sink: Sink

    Permalink
    Attributes
    protected
  13. abstract def sources: Seq[Source]

    Permalink
    Attributes
    protected
  14. abstract def sparkSession: SparkSession

    Permalink
    Attributes
    protected
  15. abstract def triggerClock: Clock

    Permalink
    Attributes
    protected

Concrete Value Members

  1. final def !=(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int

    Permalink
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0

    Permalink
    Definition Classes
    Any
  5. def clone(): AnyRef

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  6. var currentStatus: StreamingQueryStatus

    Permalink
    Attributes
    protected
  7. final def eq(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  8. def equals(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  9. def finalize(): Unit

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  10. def finishTrigger(hasNewData: Boolean): Unit

    Permalink

    Finalizes the query progress and adds it to list of recent status updates.

    Finalizes the query progress and adds it to list of recent status updates.

    Attributes
    protected
  11. final def getClass(): Class[_]

    Permalink
    Definition Classes
    AnyRef → Any
  12. def hashCode(): Int

    Permalink
    Definition Classes
    AnyRef → Any
  13. def initializeLogIfNecessary(isInterpreter: Boolean): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  14. final def isInstanceOf[T0]: Boolean

    Permalink
    Definition Classes
    Any
  15. def isTraceEnabled(): Boolean

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  16. def lastProgress: StreamingQueryProgress

    Permalink

    Returns the most recent query progress update or null if there were no progress updates.

  17. def log: Logger

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  18. def logDebug(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  19. def logDebug(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  20. def logError(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  21. def logError(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  22. def logInfo(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  23. def logInfo(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  24. def logName: String

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  25. def logTrace(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  26. def logTrace(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  27. def logWarning(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  28. def logWarning(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  29. final def ne(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  30. final def notify(): Unit

    Permalink
    Definition Classes
    AnyRef
  31. final def notifyAll(): Unit

    Permalink
    Definition Classes
    AnyRef
  32. def recentProgress: Array[StreamingQueryProgress]

    Permalink

    Returns an array containing the most recent query progress updates.

  33. def reportTimeTaken[T](triggerDetailKey: String)(body: ⇒ T): T

    Permalink

    Records the duration of running body for the next query progress update.

    Records the duration of running body for the next query progress update.

    Attributes
    protected
  34. def startTrigger(): Unit

    Permalink

    Begins recording statistics about query progress for a given trigger.

    Begins recording statistics about query progress for a given trigger.

    Attributes
    protected
  35. def status: StreamingQueryStatus

    Permalink

    Returns the current status of the query.

  36. final def synchronized[T0](arg0: ⇒ T0): T0

    Permalink
    Definition Classes
    AnyRef
  37. def toString(): String

    Permalink
    Definition Classes
    AnyRef → Any
  38. def updateStatusMessage(message: String): Unit

    Permalink

    Updates the message returned in status.

    Updates the message returned in status.

    Attributes
    protected
  39. final def wait(): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  40. final def wait(arg0: Long, arg1: Int): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  41. final def wait(arg0: Long): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Inherited from Logging

Inherited from AnyRef

Inherited from Any

Ungrouped