16. RDMA Multipathing#
cuObjClient uses the JSON configuration selected by the
CUFILE_ENV_PATH_JSON environment variable. For example:
CUFILE_ENV_PATH_JSON=/path/to/cuobj.json
Add RDMA multipathing settings under the properties object in the JSON
file. The rdma_dev_addr_list field lists the client-side RDMA device
addresses or interfaces available to cuObjClient. Configure multiple entries to
make multiple RDMA devices available for multipathing.
{
"properties": {
"rdma_dev_addr_list": [
"192.168.1.10",
"192.168.1.11",
"fd00::10",
"fd00::11"
],
"rdma_peer_type": "dmabuf",
"rdma_multipath_enabled": true,
"rdma_max_backup_devices": 2,
"rdma_io_retry_count": 3,
"rdma_io_retry_delay_ms": 100,
"rdma_failback_enabled": true,
"rdma_failback_delay_ms": 100,
"rdma_health_check_interval_ms": 100,
"rdma_async_event_monitoring": true,
"rdma_unhealthy_threshold": 3
}
}
rdma_peer_type is shown for completeness and is described in
RDMA Peer Memory Configuration.
rdma_dev_addr_list accepts JSON strings. IPv6 addresses are specified as
plain IPv6 string literals, such as "fd00::10". Do not wrap IPv6 addresses
in square brackets in this field. For link-local IPv6 addresses, include the
zone identifier in the string, for example "fe80::1%mlx5_0".
RDMA device or interface names may be used instead of IP addresses:
"rdma_dev_addr_list": ["mlx5_0", "mlx5_1"]
IP addresses and RDMA device or interface names may also be mixed:
"rdma_dev_addr_list": ["192.168.1.10", "mlx5_1", "fd00::11"]
RDMA multipathing fields:
Field |
Type |
Default |
Values |
Description |
|---|---|---|---|---|
|
array of strings |
empty |
IPv4 address, IPv6 address, or RDMA device/interface name strings |
Client-side RDMA device address list. Multiple entries make multiple RDMA devices available for multipathing. |
|
boolean |
|
|
Enables or disables RDMA multipathing. |
|
unsigned integer |
|
|
Maximum number of backup RDMA devices. Values above |
|
unsigned integer |
|
|
Number of I/O retries on retryable errors before the operation gives up.
Values above |
|
unsigned integer |
|
|
Delay, in milliseconds, between I/O retries. Values above |
|
boolean |
|
|
Enables or disables automatic failback to the primary RDMA device when it recovers. |
|
unsigned integer |
|
|
Minimum delay, in milliseconds, after failover before failback is
attempted. Values above |
|
unsigned integer |
|
|
Health check polling interval, in milliseconds. Values below |
|
boolean |
|
|
Enables or disables async RDMA event monitoring for failure detection. |
|
unsigned integer |
|
|
Number of consecutive health check failures before a device is marked
unhealthy. Values below |
When RDMA multipathing is enabled, at least two active RDMA devices are required for failover.
16.1. How RDMA Multipathing Is Used#
When rdma_multipath_enabled is set to true, cuObjClient can use the
RDMA devices listed in rdma_dev_addr_list for GET and PUT operations. One
device is used as the active path. If a retryable I/O error is reported and
another configured RDMA device is available, a retry can continue on the active
path or use another path.
rdma_io_retry_count controls the number of retry attempts.
rdma_io_retry_delay_ms controls the delay between retry attempts when the
active path is not changed. rdma_failback_enabled allows the operation path
to return to the primary device after it recovers, subject to
rdma_failback_delay_ms.
16.2. Callback Return Values That Can Trigger Retry#
GET and PUT callbacks should return the number of bytes transferred on success.
On failure, callbacks may return a negative errno value. The following
callback return values are treated as retryable:
Callback return value |
Meaning |
|---|---|
|
Operation not permitted. |
|
Connection timed out. |
|
Connection reset. |
|
Network unreachable. |
|
Host unreachable. |
|
Connection refused. |
|
Network is down. |
|
No buffer space available. |
|
Try again. |
|
Operation would block. |
|
Interrupted operation. |
|
I/O error. |
|
Device unavailable. |
|
Link severed. |
|
Communication error. |
|
Protocol error. |
|
Permission denied. |
|
Endpoint not connected. |
|
Connection aborted. |
Other negative callback return values are not retryable. -ENOTSUP is
returned without retry.