7.78. CUhostCommunicationsVtable

Defined in /dvs/p4/build/sw/gpgpu/bin/x86_64_Linux_release/stub/cuda.h

struct CUhostCommunicationsVtable

Virtual function table for communications operations, passed to some APIs to coordinate with other CUDA processes.

A communicator object (passed as this_ to the methods in this vtable) (hereafter just object) performs communication operations and may for example contain MPI communicator handles and forward certain methods to MPI. CUDA itself has no visibility into the object or any associated topology of e.g. an MPI communicator, and the object itself implicitly defines the topology of operations it is passed to at the CUDA API level. This is analgous for example to how the MPI group for a collective is implicitly defined by the communicator handle and not separately passed to APIs.

An object manages 1 or more independent communicators. CUDA will issue operations on a given communicator in a well-defined order. In order to maintain deterministic operation order on a given communicator, the same object or objects wrapping the same external communicator handles should not be passed to API calls in multiple threads that may race. Calls to the resize method are considered operations on communicator 0 and ordered accordingly. In addition, CUDA may retain a reference to an object and continue to issue operations asynchronously. As long as API calls are issued in a defined order, CUDA will maintain necessary ordering, but components outside of CUDA should not issue operations on any underlying communicator when it may be in use by CUDA (either an API call is in progress or a reference is outstanding).

All methods should return 0 for success, a negative value defined in CUcommsStatus if applicable, or an implementation-defined positive value. Positive values will be fatal to the attempted CUDA operation for which the call is being issued.

Public Members

int version

Indicates an interface revision for binary compatibility.

Set this to 1.

int (*resize)(void *this_, int size)

Resize this_ to have at least size independent communicators.

This method may be called multiple times and should be idempotent for a given value of size. Calls to this method will be deterministically ordered with initiation of operations on communicator 0.

int (*get_comm_id)(void *this_, int comm_index, cuuint64_t *id_out)

Obtain an opaque identifier for an underlying communicator.

Whenever communicators from two different objects return the same ID, so long as the associated API calls happen in a defined order, CUDA will issue operations to the communicator in deterministic order. It is permissible (but undesirable) for two distinct communicators to have an ID collision, and IDs do not need to be consistent across ranks. It is therefore typically permissible to type pun an underlying handle.

int (*barrier)(void *this_, int comm_index, void **request_out)

Issue an asynchronous barrier operation on the given comm_index.

A request object is returned for use with test or testany.

int (*allreduce_uint64)(void *this_, int comm_index, const cuuint64_t *sendbuf, cuuint64_t *recvbuf, int count, CUcommsReductionOp op, void **request_out)

Issue an asynchronous uint64_t allreduce operation on the given comm_index.

A request object is returned for use with test or testany.

int (*test)(void *this_, void **request, int *flag_out)

Check the status of an asynchronous operation.

The value of flag_out is set to non-zero if the operation is complete, in which case the request object is also deallocated and the handle value may be overwritten.

int (*testany)(void *this_, int count, void **requests, int *index_out)

Check if any of a set of operations is complete.

If so, the value of index_out is set to the corresponding request index, in which case the request object is also deallocated and the handle value may be overwritten. If none of the operations is complete, index_out is set to -1.

int (*check_async_error)(void *this_)

A nonzero return value indicates that in progress operations using this object, such as kernel launches, should be aborted.

This must be used to prevent threads from waiting indefinitely for other threads at local (within-process) portions of an operation, where there is no inter-process request object to poll.

void (*retain)(void *this_)

Retain a reference to the object.

CUDA may hold references and issue operations asynchronously beyond an API call to which the object was passed.

void (*release)(void *this_)

Release a reference to the object.