broadcast

Type Members

1. abstract class Broadcast[T] extends Serializable with Logging

   

   A broadcast variable.

   A broadcast variable. Broadcast variables allow the programmer to keep a read-only variable cached on each machine rather than shipping a copy of it with tasks. They can be used, for example, to give every node a copy of a large input dataset in an efficient manner. Spark also attempts to distribute broadcast variables using efficient broadcast algorithms to reduce communication cost.

   Broadcast variables are created from a variable v by calling org.apache.spark.SparkContext#broadcast. The broadcast variable is a wrapper around v, and its value can be accessed by calling the value method. The interpreter session below shows this:

   scala> val broadcastVar = sc.broadcast(Array(1, 2, 3))
   broadcastVar: org.apache.spark.broadcast.Broadcast[Array[Int]] = Broadcast(0)
   
   scala> broadcastVar.value
   res0: Array[Int] = Array(1, 2, 3)

   After the broadcast variable is created, it should be used instead of the value v in any functions run on the cluster so that v is not shipped to the nodes more than once. In addition, the object v should not be modified after it is broadcast in order to ensure that all nodes get the same value of the broadcast variable (e.g. if the variable is shipped to a new node later).

   T

   Type of the data contained in the broadcast variable.

2. trait BroadcastFactory extends AnyRef

   

   :: DeveloperApi :: An interface for all the broadcast implementations in Spark (to allow multiple broadcast implementations).

   :: DeveloperApi :: An interface for all the broadcast implementations in Spark (to allow multiple broadcast implementations). SparkContext uses a user-specified BroadcastFactory implementation to instantiate a particular broadcast for the entire Spark job.

   Annotations

   @DeveloperApi()

3. class HttpBroadcastFactory extends BroadcastFactory

   

   A org.apache.spark.broadcast.BroadcastFactory implementation that uses a HTTP server as the broadcast mechanism.

   A org.apache.spark.broadcast.BroadcastFactory implementation that uses a HTTP server as the broadcast mechanism. Refer to org.apache.spark.broadcast.HttpBroadcast for more details about this mechanism.

4. class TorrentBroadcastFactory extends BroadcastFactory

   

   A org.apache.spark.broadcast.Broadcast implementation that uses a BitTorrent-like protocol to do a distributed transfer of the broadcasted data to the executors.

   A org.apache.spark.broadcast.Broadcast implementation that uses a BitTorrent-like protocol to do a distributed transfer of the broadcasted data to the executors. Refer to org.apache.spark.broadcast.TorrentBroadcast for more details.