6.48. subgrid management

A subgrid schedule provides a simple scheme for avoiding a full barrier between task switches on the GPU.

It does so by allowing users to split up a kernel grid of work into subgrids for finer grained dependency management.

By organizing workItems and dependency tracking with subgrids, subsequent kernels can start right away and incrementally wait for work done by prior kernels to be finished, rather than having to wait for the prior kernel to complete all of its work before using any of it.

While PDL can achieve similar schedules by explicitly splitting grids into smaller grids, such a schedule would need to pay the startup and exit cost for CTAs in each split. In a subgrid schedule a CTA can stride across multiple subgrids of data, amortizing CTA startup and completion costs.

The subgrid API builds a schedule hierarchically:

create a schedule;
add one or more worker grids that define the worker launch configuration;
add worksets to each worker grid to group subgrids that share a user-data payload;
add subgrids to each workset and optionally attach dependencies on earlier subgrids.

The current scheduling policy is deterministic and append-ordered. Worker grids are considered in creation order. Within a worker grid, worksets are processed in creation order. Within a workset, subgrids are processed in creation order.

Subgrid schedules must be included in a cuda graph and cannot be modified after graph instantiation. The schedule is run by launching the corresponding instantiated graph.

Functions

CUresult cuSubgridCreate(CUsubgrid *subgrid_out, CUsubgridWorkset hWorkset, const CUsubgrid *hDependencies, cuuint32_t numDependencies, CUoffset3D *workItemStart, CUoffset3D *workItemEnd): create a subgrid
CUresult cuSubgridCreate_v2(CUsubgrid *subgrid_out, CUsubgridWorkset hWorkset, const CUsubgrid *hDependencies, const CUsubgridDependencyType *dependencyTypes, cuuint32_t numDependencies, CUoffset3D *workItemStart, CUoffset3D *workItemEnd): create a subgrid with dependency type support
CUresult cuSubgridWorkerGridCreate(CUsubgridWorkerGrid *workerGrid_out, CUsubgridSchedule hSchedule, const CUsubgridWorkerGridDesc *workerGridDesc): create a subgrid worker grid
CUresult cuSubgridWorksetCreate(CUsubgridWorkset *workset_out, CUsubgridWorkerGrid hWorkerGrid, const void *userData, size_t userDataSize): create a subgrid workset

6.48.1. Functions

CUresult cuSubgridCreate(CUsubgrid *subgrid_out, CUsubgridWorkset hWorkset, const CUsubgrid *hDependencies, cuuint32_t numDependencies, CUoffset3D *workItemStart, CUoffset3D *workItemEnd)

create a subgrid

A subgrid is the fundametal dependency management object in the subgrid schedule. Subgrid aware kernels will call cudaGetNextWorkItem to get items from the subgrid starting at workItemStart. Subgrid aware kernels will call cudaWaitWorkItem to ensure the associated subgrid’s dependencies are met before reading the related data. Each work item in a subgrid will be fetched and processed by only one worker (CTA/CGA) in the work group.

The application must not add dependencies to subgrids belonging to worker grids created after the worker grid the subgrid is joining. The application must not add dependencies to subgrids belonging to worksets from the owning worker grid created after the owning workset.

The easiest way to meet the dependency ordering requirement is to only add subgrid objects to the last created parent object. That is only add CUsubgridWorksets to the last created CUsubgridWorkerGrid and only add CUsubgrids to the last created CUsubgridWorkset.