Hooks that will broadcast all global variables from root rank to all other processes during initialization.
Hooks that will broadcast all global variables from root rank to all other processes during initialization.
This is necessary to ensure consistent initialization of all workers when training is started with random weights or restored from a checkpoint.
Performs an all-reduce operation on value
.
Performs an all-reduce operation on value
.
This function performs a bandwidth-optimal ring all-reduce on the input tensor. If the input is indexed slices, then this function instead does an all-gather on the values and the indices, effectively doing an all-reduce on the represented tensor.
Value to reduce.
If true
, the average over all ranks will be computed.
Device to use for dense tensor reduce operations. Defaults to a GPU if Horovod was built
with HOROVOD_GPU_ALLREDUCE
.
Device to use for sparse tensor reduce operations. Defaults to a GPU if Horovod was built
with HOROVOD_GPU_ALLGATHER
.
Reduced tensor value.
Broadcasts all global variables from root rank to all other processes.
Broadcasts all global variables from root rank to all other processes.
Rank of the process from which the global variable values will be broadcasted to all other processes.
Created broadcast op.
Initializes Horovod.
Initializes Horovod.
Returns the local Horovod rank of the calling process, within the node that it is running on.
Returns the local Horovod rank of the calling process, within the node that it is running on. For example, if there are seven processes running on a node, their local ranks will be zero through six, inclusive.
IllegalStateException
If Horovod has not been initialized yet.
Returns the number of Horovod processes within the node the current process is running on.
Returns the number of Horovod processes within the node the current process is running on.
IllegalStateException
If Horovod has not been initialized yet.
Returns a flag indicating whether MPI multi-threading is supported.
Returns a flag indicating whether MPI multi-threading is supported. If MPI multi-threading is supported, users may mix and match Horovod usage with other MPI libraries.
IllegalStateException
If Horovod has not been initialized yet.
Returns the Horovod rank of the calling process.
Returns the Horovod rank of the calling process.
IllegalStateException
If Horovod has not been initialized yet.
Returns the number of Horovod processes.
Returns the number of Horovod processes.
IllegalStateException
If Horovod has not been initialized yet.