Is this page helpful?

Multi-Process support - cuTENSORMp (Beta)#

cuTENSORMp is a multi-process extension of cuTENSOR for distributed tensor contractions. It coordinates tensor computations across multiple processes while reusing cuTENSOR’s single-process kernels for local work.

The overall programming model closely follows cuTENSOR:

  1. Create and manage a cuTENSORMp handle,

  2. Create distributed tensor descriptors and an operation descriptor,

  3. Create an execution plan using a plan preference and workspace budget,

  4. Query the required workspace, and

  5. Launch the distributed contraction.

This document summarizes the supported operation, distribution model, workspace behavior, accuracy considerations, CUDA Graph considerations, and logging controls for cuTENSORMp.

Supported operation#

cuTENSORMp currently supports distributed tensor contractions of the form

\[D = \alpha \cdot \mathrm{contract}(A, B) + \beta \cdot C .\]

The contraction is specified with mode arrays passed to cutensorMpCreateContraction(). Mode labels identify how tensor modes are matched between \(A\), \(B\), \(C\), and \(D\).

Nonzero \(\beta\) is supported. When \(\beta\) is nonzero, the \(C\) and \(D\) descriptors must describe identical layouts. The \(C\) and \(D\) pointers may be the same allocation or different allocations, provided the layouts match.

Tensor distributions and mode order#

Tensor descriptors describe the global extent, strides, block sizes, block strides, rank counts per mode, and participating ranks for each tensor. cuTENSORMp supports replicated and distributed tensors, including common block-distributed layouts across multiple tensor modes.

Mode labels are matched by name. The output mode order does not need to follow a single canonical pattern, and interleaved output modes are supported for validated distribution patterns.

Some descriptor, distribution, and mode-order combinations are outside the current support envelope. In those cases, plan creation returns CUTENSOR_STATUS_NOT_SUPPORTED rather than selecting an unsafe execution plan. Applications should treat this as a planning result and choose a different distribution, a larger workspace budget, or a fallback path.

Workspace and planning#

cutensorMpCreatePlanPreference() accepts a device and host workspace budget. The planner uses this budget when selecting an execution plan. After a plan is created, query CUTENSORMP_PLAN_REQUIRED_WORKSPACE_DEVICE and CUTENSORMP_PLAN_REQUIRED_WORKSPACE_HOST with cutensorMpPlanGetAttribute() and pass workspace allocations of at least those sizes to cutensorMpContract().

The device workspace includes both cuTENSORMp communication buffers and the local cuTENSOR workspace reserved by the selected plan. A smaller workspace budget can cause the planner to choose a more memory-conservative plan, and a budget below the required lower bound can cause plan creation to fail.

Performance guidelines#

The same general performance guidance as cuTENSOR applies to local contractions: keep local tensor tiles large enough for efficient kernels and prefer layouts with good memory locality. For distributed execution, also consider the following:

  • Choose tensor distributions that expose enough parallelism across ranks.

  • Avoid very small local tiles when possible.

  • Provide enough workspace for the planner to use communication-efficient plans.

  • Query the planned workspace and reuse it across repeated executions of the same plan.

Performance depends on the tensor shapes, distribution, data type, workspace budget, and system topology.

Accuracy guarantees#

cuTENSORMp uses cuTENSOR kernels for local contractions and follows cuTENSOR’s data type and compute descriptor rules. Distributed execution may change the order in which partial results are accumulated, so results should be compared with an appropriate numerical tolerance rather than bitwise equality.

Scalar types follow the same scalar type rules as cuTENSOR for the selected data type and compute descriptor.

CUDA Graph Support#

cuTENSORMp orchestrates device-side cuTENSOR kernels and inter-process communication. End-to-end cuTENSORMp contractions are not guaranteed to be CUDA-graph-capturable, because communication and host-side coordination may be present in the execution path.

If CUDA Graph capture is required, verify capture behavior for the exact operation, distribution, and runtime environment being used.

Logging#

cuTENSORMp logging can be configured with environment variables before launching the application:

  • CUTENSORMP_LOG_LEVEL enables logging and controls the logging level.

  • CUTENSORMP_LOG_MASK is an alternative mask-based filter.

  • CUTENSORMP_LOG_FILE writes enabled logs to a file instead of standard output. In multi-process runs, use a process-specific file name pattern to avoid multiple ranks writing the same file.

  • CUTENSORMP_LOG_RANK restricts log output to a single rank.