16. RDMA Multipathing#

cuObjClient uses the JSON configuration selected by the CUFILE_ENV_PATH_JSON environment variable. For example:

CUFILE_ENV_PATH_JSON=/path/to/cuobj.json

Add RDMA multipathing settings under the properties object in the JSON file. The rdma_dev_addr_list field lists the client-side RDMA device addresses or interfaces available to cuObjClient. Configure multiple entries to make multiple RDMA devices available for multipathing.

{
  "properties": {
    "rdma_dev_addr_list": [
      "192.168.1.10",
      "192.168.1.11",
      "fd00::10",
      "fd00::11"
    ],
    "rdma_peer_type": "dmabuf",
    "rdma_multipath_enabled": true,
    "rdma_max_backup_devices": 2,
    "rdma_io_retry_count": 3,
    "rdma_io_retry_delay_ms": 100,
    "rdma_failback_enabled": true,
    "rdma_failback_delay_ms": 100,
    "rdma_health_check_interval_ms": 100,
    "rdma_async_event_monitoring": true,
    "rdma_unhealthy_threshold": 3
  }
}

rdma_peer_type is shown for completeness and is described in RDMA Peer Memory Configuration.

rdma_dev_addr_list accepts JSON strings. IPv6 addresses are specified as plain IPv6 string literals, such as "fd00::10". Do not wrap IPv6 addresses in square brackets in this field. For link-local IPv6 addresses, include the zone identifier in the string, for example "fe80::1%mlx5_0".

RDMA device or interface names may be used instead of IP addresses:

"rdma_dev_addr_list": ["mlx5_0", "mlx5_1"]

IP addresses and RDMA device or interface names may also be mixed:

"rdma_dev_addr_list": ["192.168.1.10", "mlx5_1", "fd00::11"]

RDMA multipathing fields:

Field

Type

Default

Values

Description

rdma_dev_addr_list

array of strings

empty

IPv4 address, IPv6 address, or RDMA device/interface name strings

Client-side RDMA device address list. Multiple entries make multiple RDMA devices available for multipathing.

rdma_multipath_enabled

boolean

false

true or false

Enables or disables RDMA multipathing.

rdma_max_backup_devices

unsigned integer

2

0 to 8

Maximum number of backup RDMA devices. Values above 8 are capped at 8.

rdma_io_retry_count

unsigned integer

3

0 to 10

Number of I/O retries on retryable errors before the operation gives up. Values above 10 are capped at 10.

rdma_io_retry_delay_ms

unsigned integer

100

0 to 10000

Delay, in milliseconds, between I/O retries. Values above 10000 are capped at 10000.

rdma_failback_enabled

boolean

true

true or false

Enables or disables automatic failback to the primary RDMA device when it recovers.

rdma_failback_delay_ms

unsigned integer

100

0 to 60000

Minimum delay, in milliseconds, after failover before failback is attempted. Values above 60000 are capped at 60000.

rdma_health_check_interval_ms

unsigned integer

100

100 to 60000

Health check polling interval, in milliseconds. Values below 100 are raised to 100. Values above 60000 are capped at 60000.

rdma_async_event_monitoring

boolean

true

true or false

Enables or disables async RDMA event monitoring for failure detection.

rdma_unhealthy_threshold

unsigned integer

3

1 to 100

Number of consecutive health check failures before a device is marked unhealthy. Values below 1 are raised to 1. Values above 100 are capped at 100.

When RDMA multipathing is enabled, at least two active RDMA devices are required for failover.

16.1. How RDMA Multipathing Is Used#

When rdma_multipath_enabled is set to true, cuObjClient can use the RDMA devices listed in rdma_dev_addr_list for GET and PUT operations. One device is used as the active path. If a retryable I/O error is reported and another configured RDMA device is available, a retry can continue on the active path or use another path.

rdma_io_retry_count controls the number of retry attempts. rdma_io_retry_delay_ms controls the delay between retry attempts when the active path is not changed. rdma_failback_enabled allows the operation path to return to the primary device after it recovers, subject to rdma_failback_delay_ms.

16.2. Callback Return Values That Can Trigger Retry#

GET and PUT callbacks should return the number of bytes transferred on success. On failure, callbacks may return a negative errno value. The following callback return values are treated as retryable:

Callback return value

Meaning

-EPERM

Operation not permitted.

-ETIMEDOUT

Connection timed out.

-ECONNRESET

Connection reset.

-ENETUNREACH

Network unreachable.

-EHOSTUNREACH

Host unreachable.

-ECONNREFUSED

Connection refused.

-ENETDOWN

Network is down.

-ENOBUFS

No buffer space available.

-EAGAIN

Try again.

-EWOULDBLOCK

Operation would block.

-EINTR

Interrupted operation.

-EIO

I/O error.

-ENODEV

Device unavailable.

-ENOLINK

Link severed.

-ECOMM

Communication error.

-EPROTO

Protocol error.

-EACCES

Permission denied.

-ENOTCONN

Endpoint not connected.

-ECONNABORTED

Connection aborted.

Other negative callback return values are not retryable. -ENOTSUP is returned without retry.