Skip to navigation

INFINIA

View as Markdown

Overview

The INFINIA backend connects NIXL to DDN INFINIA object storage. It uses the INFINIA Client and its C++20 coroutine-based asynchronous API to read and write objects directly from registered host or GPU memory.

PropertyValue
Transfer TypesDRAM ↔ Object, VRAM ↔ Object
TechnologyDDN INFINIA Client and Async libraries
Best ForCheckpoints and other large-object workflows on DDN INFINIA

Installation

The backend requires an INFINIA installation containing these libraries and headers:

  • libred_async.so (Infinia SDK 2.5.0 or later)
  • red/red_async.hpp
  • red/red_status.h

Configure NIXL with the path to the INFINIA installation:

meson setup build -Dinfinia_path=/path/to/infinia
meson compile -C build

The path must contain lib/ and include/ directories.

Configuration

Create the backend with parameters passed to createBackend("INFINIA", params):

ParameterDescriptionDefault
clusterINFINIA cluster namecluster1
tenantTenant namered
subtenantSubtenant namered
datasetDataset namenixl
sthreadsNumber of service threads8
num_buffersNumber of operation buffers512
num_ring_entriesNumber of ring entries512
coremasksCPU affinity mask0x2
use_dmabufUse DMA-BUF for GPU memory registrationtrue
max_retriesMaximum retries for failed operationsLibrary default
batch_sizeAsync operations per batchLibrary default

The RED_CLUSTER, RED_TENANT, and RED_DATASET environment variables override backend parameters. RED_TENANT may include a subtenant as tenant/subtenant.

nixl_b_params_t params = {
{"cluster", "mycluster"},
{"tenant", "mytenant"},
{"subtenant", "mysubtenant"},
{"dataset", "mydataset"},
};
agent.createBackend("INFINIA", params);

Alternatively, use NIXL’s common TOML configuration by pointing NIXL_CONFIG_FILE at a file such as:

RED_CLUSTER = "mycluster"
RED_TENANT = "mytenant/mysubtenant"
RED_DATASET = "mydataset"
[infinia]
sthreads = 8
num_buffers = 512
num_ring_entries = 512
coremasks = "0x2"
export NIXL_CONFIG_FILE=/path/to/nixl-infinia.toml

Values in the [infinia] table apply only to settings not passed as backend parameters.

Memory Registration

Register local buffers as DRAM_SEG or VRAM_SEG. Register the remote object as OBJ_SEG; its descriptor metaInfo is used as the object key. If metaInfo is empty, the backend uses the object descriptor’s device ID as the key.

Transfers are asynchronous. Poll them with checkXfer, and release completed request handles with releaseReqH.