Class

org.apache.spark.sql.execution.streaming

FileStreamSource

Related Doc: package streaming

Permalink

class FileStreamSource extends Source with Logging

A very simple source that reads text files from the given directory as they appear.

TODO Clean up the metadata files periodically

Linear Supertypes
Logging, Source, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. FileStreamSource
  2. Logging
  3. Source
  4. AnyRef
  5. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new FileStreamSource(sparkSession: SparkSession, path: String, fileFormatClassName: String, schema: StructType, metadataPath: String, options: Map[String, String])

    Permalink

Value Members

  1. final def !=(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any

  2. final def ##(): Int

    Permalink
    Definition Classes
    AnyRef → Any

  3. final def ==(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any

  4. final def asInstanceOf[T0]: T0

    Permalink
    Definition Classes
    Any

  5. def clone(): AnyRef

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  6. def currentOffset: LongOffset

    Permalink

    Return the latest offset in the source

  7. final def eq(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef

  8. def equals(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any

  9. def finalize(): Unit

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )

  10. def getBatch(start: Option[Offset], end: Offset): DataFrame

    Permalink

    Returns the data that is between the offsets (start, end].

    Returns the data that is between the offsets (start, end].

    Definition Classes
    FileStreamSource → Source

  11. final def getClass(): Class[_]

    Permalink
    Definition Classes
    AnyRef → Any

  12. def getOffset: Option[Offset]

    Permalink

    Returns the maximum available offset for this source.

    Returns the maximum available offset for this source.

    Definition Classes
    FileStreamSource → Source

  13. def hashCode(): Int

    Permalink
    Definition Classes
    AnyRef → Any

  14. def initializeLogIfNecessary(isInterpreter: Boolean): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  15. final def isInstanceOf[T0]: Boolean

    Permalink
    Definition Classes
    Any

  16. def isTraceEnabled(): Boolean

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  17. def log: Logger

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  18. def logDebug(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  19. def logDebug(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  20. def logError(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  21. def logError(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  22. def logInfo(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  23. def logInfo(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  24. def logName: String

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  25. def logTrace(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  26. def logTrace(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  27. def logWarning(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  28. def logWarning(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  29. final def ne(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef

  30. final def notify(): Unit

    Permalink
    Definition Classes
    AnyRef

  31. final def notifyAll(): Unit

    Permalink
    Definition Classes
    AnyRef

  32. val schema: StructType

    Permalink

    Returns the schema of the data from this source

    Returns the schema of the data from this source

    Definition Classes
    FileStreamSource → Source

  33. def stop(): Unit

    Permalink

    Stop this source and free any resources it has allocated.

    Stop this source and free any resources it has allocated.

    Definition Classes
    FileStreamSource → Source

  34. final def synchronized[T0](arg0: ⇒ T0): T0

    Permalink
    Definition Classes
    AnyRef

  35. def toString(): String

    Permalink
    Definition Classes
    FileStreamSource → AnyRef → Any

  36. final def wait(): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  37. final def wait(arg0: Long, arg1: Int): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  38. final def wait(arg0: Long): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  39. def withBatchingLocked[T](func: ⇒ T): T

    Permalink

    For test only.

    For test only. Run func with the internal lock to make sure when func is running, the current offset won't be changed and no new batch will be emitted.

Inherited from Logging

Inherited from Source

Inherited from AnyRef

Inherited from Any

Ungrouped