Class

org.apache.spark.sql.execution.streaming.FileStreamSource

SeenFilesMap

Related Doc: package FileStreamSource

Permalink

class SeenFilesMap extends AnyRef

A custom hash map used to track the list of files seen. This map is not thread-safe.

To prevent the hash map from growing indefinitely, a purge function is available to remove files "maxAgeMs" older than the latest file.

Linear Supertypes
AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. SeenFilesMap
  2. AnyRef
  3. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new SeenFilesMap(maxAgeMs: Long)

    Permalink

Value Members

  1. final def !=(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any

  2. final def ##(): Int

    Permalink
    Definition Classes
    AnyRef → Any

  3. final def ==(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any

  4. def add(path: String, timestamp: Timestamp): Unit

    Permalink

    Add a new file to the map.

  5. def allEntries: Seq[(String, Timestamp)]

    Permalink

  6. final def asInstanceOf[T0]: T0

    Permalink
    Definition Classes
    Any

  7. def clone(): AnyRef

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  8. final def eq(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef

  9. def equals(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any

  10. def finalize(): Unit

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )

  11. final def getClass(): Class[_]

    Permalink
    Definition Classes
    AnyRef → Any

  12. def hashCode(): Int

    Permalink
    Definition Classes
    AnyRef → Any

  13. final def isInstanceOf[T0]: Boolean

    Permalink
    Definition Classes
    Any

  14. def isNewFile(path: String, timestamp: Timestamp): Boolean

    Permalink

    Returns true if we should consider this file a new file.

    Returns true if we should consider this file a new file. The file is only considered "new" if it is new enough that we are still tracking, and we have not seen it before.

  15. final def ne(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef

  16. final def notify(): Unit

    Permalink
    Definition Classes
    AnyRef

  17. final def notifyAll(): Unit

    Permalink
    Definition Classes
    AnyRef

  18. def purge(): Int

    Permalink

    Removes aged entries and returns the number of files removed.

  19. def size: Int

    Permalink

  20. final def synchronized[T0](arg0: ⇒ T0): T0

    Permalink
    Definition Classes
    AnyRef

  21. def toString(): String

    Permalink
    Definition Classes
    AnyRef → Any

  22. final def wait(): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  23. final def wait(arg0: Long, arg1: Int): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  24. final def wait(arg0: Long): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Inherited from AnyRef

Inherited from Any

Ungrouped