Connector

Manage connections and create read/write operations between workers in a Dynamo graph.

View as Markdown

Core class for managing the connection between workers in a distributed environment. Use this class to create readable and writable operations, or read and write data to remote workers.

This class provides a “pythonic” interface using NIXL library to utilize GPU Direct RDMA accelerated, when available, data transfers between models hosted by different workers in a Dynamo graph. The connector provides two methods of moving data between workers:

  • Preparing local memory to be written to by a remote worker.

  • Preparing local memory to be read by a remote worker.

In both cases, local memory is registered with the NIXL-based I/O subsystem via the Descriptor class and provided to the connector. When RDMA is available, the connector then configures the RDMA subsystem to expose the memory for the requested operation and returns an operation control object; otherwise the connector will select the best available RDMA alternative. The operation control object, either a ReadableOperation or a WritableOperation, provides NIXL metadata (RdmaMetadata) via its .metadata() method, functionality to query the operation’s current state, as well as the ability to cancel the operation prior to its completion.

The NIXL metadata must be provided to the remote worker expected to complete the operation. The metadata contains required information (identifiers, keys, etc.) which enables the remote worker to interact with the provided memory.

NIXL metadata contains a worker’s address as well as security keys to access specific registered memory descriptors. This data provides direct memory access between workers, and should be considered sensitive and therefore handled accordingly.

Example Usage

1    @async_on_start
2    async def async_init(self):
3      self.connector = dynamo.nixl_connect.Connector()

See ReadOperation, ReadableOperation, WritableOperation, and WriteOperation for additional examples.

Methods

begin_read

1async def begin_read(
2    self,
3    remote_metadata: RdmaMetadata,
4    local_descriptors: Descriptor | list[Descriptor],
5) -> ReadOperation

Creates a ReadOperation for transferring data from a remote worker.

remote_metadata
RdmaMetadataRequired

Serialized request from a remote worker’s ReadableOperation. Must be transferred from the remote to the local worker via a secondary channel, most likely HTTP or TCP+NATS.

local_descriptors
Descriptor | list[Descriptor]Required

Matching set of local memory descriptors which reference memory intended to receive data from the remote worker.

Returns ReadOperation — once created, data transfer will begin immediately.

Disposal of the object will instruct the NIXL subsystem to cancel the operation, therefore the operation should be awaited until completed unless cancellation is intended. Use .wait_for_completion() to block the caller until the operation has completed or encountered an error.

begin_write

1async def begin_write(
2    self,
3    local_descriptors: Descriptor | list[Descriptor],
4    remote_metadata: RdmaMetadata,
5) -> WriteOperation

Creates a WriteOperation for transferring data to a remote worker.

local_descriptors
Descriptor | list[Descriptor]Required

Matching set of local memory descriptors which reference memory to be transferred to the remote worker.

remote_metadata
RdmaMetadataRequired

Serialized request from a remote worker’s WritableOperation. Must be transferred from the remote to the local worker via a secondary channel, most likely HTTP or TCP+NATS.

Returns WriteOperation — once created, data transfer will begin immediately.

Disposal of the object will instruct the NIXL subsystem to cancel the operation, therefore the operation should be awaited until completed unless cancellation is intended. Use .wait_for_completion() to block the caller until the operation has completed or encountered an error.

create_readable

1async def create_readable(
2    self,
3    local_descriptors: Descriptor | list[Descriptor],
4) -> ReadableOperation

Creates a ReadableOperation for transferring data to a remote worker.

local_descriptors
Descriptor | list[Descriptor]Required

Set of local memory descriptors that reference memory intended to be transferred to a remote worker.

Returns ReadableOperation — the memory referenced by the provided descriptors becomes immediately readable by a remote worker with the necessary metadata.

The metadata required to access the memory referenced by the provided descriptors is accessible via the operation’s .metadata() method. Once acquired, the metadata needs to be provided to a remote worker via a secondary channel, most likely HTTP or TCP+NATS.

Disposal of the object will instruct the NIXL subsystem to cancel the operation, therefore the operation should be awaited until completed unless cancellation is intended. Use .wait_for_completion() to block the caller until the operation has completed or encountered an error.

create_writable

1async def create_writable(
2    self,
3    local_descriptors: Descriptor | list[Descriptor],
4) -> WritableOperation

Creates a WritableOperation for transferring data from a remote worker.

local_descriptors
Descriptor | list[Descriptor]Required

Set of local memory descriptors which reference memory intended to receive data from a remote worker.

Returns WritableOperation — the memory referenced by the provided descriptors becomes immediately writable by a remote worker with the necessary metadata.

The metadata required to access the memory referenced by the provided descriptors is accessible via the operation’s .metadata() method. Once acquired, the metadata needs to be provided to a remote worker via a secondary channel, most likely HTTP or TCP+NATS.

Disposal of the object will instruct the NIXL subsystem to cancel the operation, therefore the operation should be awaited until completed unless cancellation is intended. Use .wait_for_completion() to block the caller until the operation has completed or encountered an error.

Properties

hostname
str

Gets the name of the current worker’s host.

is_cuda_available
bool

Gets True when CUDA is available for the selected array module (most likely CuPy); otherwise False.

name
str | None

Gets the Dynamo component name used by the connector.