IntelĀ® Machine Learning Scaling Library
2018
A library providing an efficient implementation of communication patterns used in deep learning.
|
A class to hold information about the parallelism scheme being used. More...
#include <mlsl.hpp>
Public Member Functions | |
size_t | GetProcessIdx (GroupType groupType) |
size_t | GetProcessCount (GroupType groupType) |
CommReq * | Bcast (void *buffer, size_t count, DataType dataType, size_t rootIdx, GroupType groupType) |
CommReq * | Reduce (void *sendBuffer, void *recvBuffer, size_t count, DataType dataType, ReductionType redType, size_t rootIdx, GroupType groupType) |
CommReq * | AllReduce (void *sendBuffer, void *recvBuffer, size_t count, DataType dataType, ReductionType redType, GroupType groupType) |
CommReq * | AlltoAll (void *sendBuffer, size_t sendCount, void *recvBuffer, DataType dataType, GroupType groupType) |
CommReq * | AlltoAllv (void *sendBuffer, size_t *sendCounts, size_t *sendOffsets, void *recvBuffer, size_t *recvCounts, size_t *recvOffsets, DataType dataType, GroupType groupType) |
CommReq * | Gather (void *sendBuffer, size_t sendCount, void *recvBuffer, DataType dataType, size_t rootIdx, GroupType groupType) |
CommReq * | AllGather (void *sendBuffer, size_t sendCount, void *recvBuffer, DataType dataType, GroupType groupType) |
CommReq * | Scatter (void *sendBuffer, void *recvBuffer, size_t recvCount, DataType dataType, size_t rootIdx, GroupType groupType) |
CommReq * | ReduceScatter (void *sendBuffer, void *recvBuffer, size_t recvCount, DataType dataType, ReductionType redType, GroupType groupType) |
void | Barrier (GroupType groupType) |
A class to hold information about the parallelism scheme being used.
Holds information about the parallelism scheme currently in use (data/model/hybrid parallelism) and provides the low-level API for collectives over data/model/global groups of processes.
These collectives can be used, for example, to broadcast initial parameters, and to collect the snapshot in model/hybrid parallelism.
CommReq* MLSL::Distribution::AllGather | ( | void * | sendBuffer, |
size_t | sendCount, | ||
void * | recvBuffer, | ||
DataType | dataType, | ||
GroupType | groupType | ||
) |
Gathers values from all processes of the group and distribute the combined values to all processes of the group.
sendBuffer | the address of the send buffer |
sendCount | the number of elements in the send buffer |
recvBuffer | the address of the receive buffer, the receive buffer should be large enough to hold messages from all processes of the group |
dataType | the data type of buffers' elements |
groupType | the group type |
CommReq* MLSL::Distribution::AllReduce | ( | void * | sendBuffer, |
void * | recvBuffer, | ||
size_t | count, | ||
DataType | dataType, | ||
ReductionType | redType, | ||
GroupType | groupType | ||
) |
Reduces values from all processes of the group and distributes the result back to all the processes.
sendBuffer | the address of the send buffer |
recvBuffer | the address of the receive buffer, meaningful only at the root process |
count | the number of elements in send/receive buffers |
dataType | the data type of buffer's elements |
redType | the reduction operation type |
groupType | the group type |
CommReq* MLSL::Distribution::AlltoAll | ( | void * | sendBuffer, |
size_t | sendCount, | ||
void * | recvBuffer, | ||
DataType | dataType, | ||
GroupType | groupType | ||
) |
Sends values from all processes of the group to all processes of the group.
sendBuffer | the address of the send buffer |
sendCount | the number of elements in the send group buffer, the send buffer should be large enough to hold messages for all processes of the group (at least (sendCount * GetProcessCount(groupType) ) elements) |
recvBuffer | the address of the receive buffer, the receive buffer should be large enough to hold messages from all processes of the group |
dataType | the data type of buffers' elements |
groupType | the group type |
CommReq* MLSL::Distribution::AlltoAllv | ( | void * | sendBuffer, |
size_t * | sendCounts, | ||
size_t * | sendOffsets, | ||
void * | recvBuffer, | ||
size_t * | recvCounts, | ||
size_t * | recvOffsets, | ||
DataType | dataType, | ||
GroupType | groupType | ||
) |
Gathers data from and scatters data to all members of a group.
sendBuffer | the address of the send buffer |
sendCounts | the number of data elements that the process sends in the buffer that is specified in the sendbuf parameter |
sendOffsets | the location, relative to the sendbuf parameter, of the data for each communicator process |
recvBuffer | the address of receive buffer the receive buffer should be large enough to hold messages from all processes of the group |
recvCounts | the number of data elements from each communicator process in the receive buffer |
recvOffsets | the location, relative to the recvbuf parameter, of the data from each communicator process |
dataType | the data type of buffers' elements |
groupType | the group type |
void MLSL::Distribution::Barrier | ( | GroupType | groupType | ) |
Sets a barrier for all the processes within a group.
groupType | the group type |
CommReq* MLSL::Distribution::Bcast | ( | void * | buffer, |
size_t | count, | ||
DataType | dataType, | ||
size_t | rootIdx, | ||
GroupType | groupType | ||
) |
Broadcasts a values from the root process to all other processes of the group.
buffer | the starting address of the buffer |
count | the number of elements in the buffer |
dataType | the data type of the buffer elements |
rootIdx | the index of the root process within a group |
groupType | group type |
CommReq* MLSL::Distribution::Gather | ( | void * | sendBuffer, |
size_t | sendCount, | ||
void * | recvBuffer, | ||
DataType | dataType, | ||
size_t | rootIdx, | ||
GroupType | groupType | ||
) |
Gathers values from all processes of the group to the root process.
sendBuffer | the address of the send buffer |
sendCount | the number of elements in the send buffer |
recvBuffer | the address of the receive buffer, meaningful only at the root process, the receive buffer should be large enough to hold messages from all processes of the group |
dataType | the data type of buffers' elements |
rootIdx | the index of the root process within a group |
groupType | the group type |
size_t MLSL::Distribution::GetProcessCount | ( | GroupType | groupType | ) |
Returns the number of processes within a group.
groupType | the group type |
size_t MLSL::Distribution::GetProcessIdx | ( | GroupType | groupType | ) |
Returns the process index within a group.
groupType | the group type |
CommReq* MLSL::Distribution::Reduce | ( | void * | sendBuffer, |
void * | recvBuffer, | ||
size_t | count, | ||
DataType | dataType, | ||
ReductionType | redType, | ||
size_t | rootIdx, | ||
GroupType | groupType | ||
) |
Reduces values on all processes within a group.
sendBuffer | the address of the send buffer |
recvBuffer | the address of the receive buffer, meaningful only at the root process |
count | the number of elements in send/receive buffers |
dataType | data the type of buffers' elements |
redType | the reduction operation type |
rootIdx | the index of the root process within a group |
groupType | the group type |
CommReq* MLSL::Distribution::ReduceScatter | ( | void * | sendBuffer, |
void * | recvBuffer, | ||
size_t | recvCount, | ||
DataType | dataType, | ||
ReductionType | redType, | ||
GroupType | groupType | ||
) |
Combines values from all processes of the group and scatters the results in blocks back to all the processes.
sendBuffer | the address of the send buffer |
recvBuffer | the address of the receive buffer |
recvCount | the number of elements in the receive buffer (the number of elements in a block) |
dataType | the data type of buffers' elements |
redType | the reduction operation type |
groupType | the group type |
CommReq* MLSL::Distribution::Scatter | ( | void * | sendBuffer, |
void * | recvBuffer, | ||
size_t | recvCount, | ||
DataType | dataType, | ||
size_t | rootIdx, | ||
GroupType | groupType | ||
) |
Sends values from the root process to all processes of the group.
sendBuffer | the address of the send buffer, meaningful only at the root process, the send buffer should be large enough to hold messages for all processes of the group |
recvBuffer | the address of the receive buffer |
recvCount | the number of elements in the receive buffer |
dataType | the data type of buffers' elements |
rootIdx | the index of the root process within a group |
groupType | the group type |