storage

Type Members

1. sealed abstract class BlockId extends AnyRef

   

   :: DeveloperApi :: Identifies a particular Block of data, usually associated with a single file.

   :: DeveloperApi :: Identifies a particular Block of data, usually associated with a single file. A Block can be uniquely identified by its filename, but each type of Block has a different set of keys which produce its unique name.

   If your BlockId should be serializable, be sure to add it to the BlockId.apply() method.

   Annotations

   @DeveloperApi()

2. class BlockManagerId extends Externalizable

   

   :: DeveloperApi :: This class represent an unique identifier for a BlockManager.

   :: DeveloperApi :: This class represent an unique identifier for a BlockManager.

   The first 2 constructors of this class is made private to ensure that BlockManagerId objects can be created only using the apply method in the companion object. This allows de-duplication of ID objects. Also, constructor parameters are private to ensure that parameters cannot be modified from outside this class.

   Annotations

   @DeveloperApi()

3. class BlockNotFoundException extends Exception

   

4. case class BlockStatus(storageLevel: StorageLevel, memSize: Long, diskSize: Long, externalBlockStoreSize: Long) extends Product with Serializable

   

   Annotations

   @DeveloperApi()

5. case class BroadcastBlockId(broadcastId: Long, field: String = "") extends BlockId with Product with Serializable

   

   Annotations

   @DeveloperApi()

6. case class RDDBlockId(rddId: Int, splitIndex: Int) extends BlockId with Product with Serializable

   

   Annotations

   @DeveloperApi()

7. class RDDInfo extends Ordered[RDDInfo]

   

   Annotations

   @DeveloperApi()

8. case class ShuffleBlockId(shuffleId: Int, mapId: Int, reduceId: Int) extends BlockId with Product with Serializable

   

   Annotations

   @DeveloperApi()

9. case class ShuffleDataBlockId(shuffleId: Int, mapId: Int, reduceId: Int) extends BlockId with Product with Serializable

   

   Annotations

   @DeveloperApi()

10. case class ShuffleIndexBlockId(shuffleId: Int, mapId: Int, reduceId: Int) extends BlockId with Product with Serializable

    

    Annotations

    @DeveloperApi()

11. class StorageLevel extends Externalizable

    

    :: DeveloperApi :: Flags for controlling the storage of an RDD.

    :: DeveloperApi :: Flags for controlling the storage of an RDD. Each StorageLevel records whether to use memory, or ExternalBlockStore, whether to drop the RDD to disk if it falls out of memory or ExternalBlockStore, whether to keep the data in memory in a serialized format, and whether to replicate the RDD partitions on multiple nodes.

    The org.apache.spark.storage.StorageLevel$ singleton object contains some static constants for commonly useful storage levels. To create your own storage level object, use the factory method of the singleton object (StorageLevel(...)).

    Annotations

    @DeveloperApi()

12. class StorageStatus extends AnyRef

    

    :: DeveloperApi :: Storage information for each BlockManager.

    :: DeveloperApi :: Storage information for each BlockManager.

    This class assumes BlockId and BlockStatus are immutable, such that the consumers of this class cannot mutate the source of the information. Accesses are not thread-safe.

    Annotations

    @DeveloperApi()

13. class StorageStatusListener extends SparkListener

    

    :: DeveloperApi :: A SparkListener that maintains executor storage status.

    :: DeveloperApi :: A SparkListener that maintains executor storage status.

    This class is thread-safe (unlike JobProgressListener)

    Annotations

    @DeveloperApi()

14. case class StreamBlockId(streamId: Int, uniqueId: Long) extends BlockId with Product with Serializable

    

    Annotations

    @DeveloperApi()

15. case class TaskResultBlockId(taskId: Long) extends BlockId with Product with Serializable

    

    Annotations

    @DeveloperApi()

16. final class TimeTrackingOutputStream extends OutputStream