7.5. Clara Pipeline Fast I/O Allocations

All Fast I/O memory resources provided by the Fast I/O Context are accessed by nvidia_clara_fastio_alloc object pointers, which act as local references to the underlying memory resource.

In addition to pointing to the underlying Fast I/O memory resources, each allocation object also contain a set of per-allocation metadata state which can be used to describe how the memory resources of that allocation are being used. For example, nvidia_clara_fastio__set_type_and_dimensions can be used to set the element type and array dimensions for a Fast I/O allocation being used to store typed, N-dimensional array data.

It’s the caller’s responsibility to call one of nvidia_clara_fastio__release or nvidia_clara_fastio__free in order to release every nvidia_clara_fastio_alloc reference that it creates or gets from the context before returning from the execute callback.

see also: Fast I/O Context

7.5.1. Allocation Ownership and Lifespan

Every nvidia_clara_fastio_alloc pointer returned by nvidia_clara_fastio__create or nvidia_clara_fastio__get acts as a reference to Fast I/O memory resources, and so each one must be released by using one of nvidia_clara_fastio__release or nvidia_clara_fastio__free.

The difference between release and free is that while both release the local reference to the underlying memory resource, free will also mark the resource for deletion, preventing any further access to the resources (though existing references and mappings will remain valid until they are also released). Freeing Fast I/O memory resources reduces the overall footprint of the pipeline job, and so allocations should be freed with nvidia_clara_fastio__free as soon as they are no longer needed by any operators in the pipeline.

However, since multiple operators in a pipeline may be consuming the same Fast I/O resource, it may not be known to the operator developer whether the Fast I/O inputs it consumes need to be freed after use or if they’ll be used again by another operator in the pipeline.

If the pipeline(s) using the operator are known to the operator developer, and they can guarantee which are the terminal operators for each allocation, then those terminal operators should also free those allocations.

If the pipeline(s) using the operator are unknown, or there is a possibility that additional operators may be added to a pipeline to consume the Fast I/O allocations, then the operator should let the underlying resource persist (but still release the local reference with nvidia_clara_fastio__release).

Note that all Fast I/O memory resources are limited to the pod that is executing the Clara Pipeline, and so all resources will be implicitly freed once the pod has completed executing.

7.5.2. Core Allocation and Memory Management Functions

The following functions are considered the core Fast I/O allocation functions, and are responsible for managing the nvidia_clara_fastio_alloc references in addition to providing functions to map the underlying memory resources for read and/or write access from the application.

7.5.2.1. Release

Releases a local nvidia_clara_fastio_alloc reference. Releasing an allocation object will implicitly unmap any previously mapped pointers to the memory resource, release its reference to the underlying resource, and then destroy the allocation object and its associated metadata.

Since this function only releases the local allocation reference, the underlying Fast I/O memory resource that it pointed to will persist within the context, and thus can still be retrieved from a context using nvidia_clara_fastio__get (assuming it’s been previous published to the context). In order to both release the local reference and also free the underlying memory resources, preventing future use of the resource, use nvidia_clara_fastio__free.

Parameters

  • alloc

    type: nvidia_clara_fastio_alloc *

    direction: In

    Pointer to the Fast I/O allocation to release.

Return

Returns zero if successful; otherwise a non-zero value.

result nvidia_clara_fastio__release(
    IN nvidia_clara_fastio_alloc *alloc
);

7.5.2.2. Free

Releases a local nvidia_clara_fastio_alloc reference and frees the underlying Fast I/O memory resource. This behaves like nvidia_clara_fastio__release, except that the underlying Fast I/O memory resource is also marked to be freed, which prevents it from being accessed or mapped by any other operator. Note that existing mappings to the data in other operators will remain valid until unmapped, but attempting to map the resource after the free will fail.

Note that Fast I/O resources may be limited, and so freeing resources as early as possible within the pipeline is recommended to minimize resource usage. If possible, allocations should be freed by the last operator in the pipeline that accesses them.

Parameters

  • alloc

    type: nvidia_clara_fastio_alloc *

    direction: In

    Pointer to the Fast I/O allocation to free.

Return

Returns zero if successful; otherwise a non-zero value.

result nvidia_clara_fastio__free(
    IN nvidia_clara_fastio_alloc *alloc
);

7.5.2.3. Are Same

Compares two nvidia_clara_fastio_alloc references and returns whether or not they point to the same underlying Fast I/O memory resource. Note that the metadata of the two objects do not need to match, just the memory resource.

Parameters

  • lhs

    type: nvidia_clara_fastio_alloc *

    direction: In

    Pointer to the first Fast I/O allocation to compare.

  • rhs

    type: nvidia_clara_fastio_alloc *

    direction: In

    Pointer to the second Fast I/O allocation to compare.

  • bool_out

    type: int *

    direction: Out

    Written with a non-zero value when the allocations point to the same memory resource, or zero if they differ.

Return

Returns zero if successful; otherwise a non-zero value.

result nvidia_clara_fastio__are_same(
    IN nvidia_clara_fastio_alloc *lhs,
    IN nvidia_clara_fastio_alloc *rhs,
    OUT int *bool_out
);

7.5.2.4. Size

Returns the size, in bytes, of the memory resource backing the nvidia_clara_fastio_alloc.

Parameters

  • alloc

    type: nvidia_clara_fastio_alloc *

    direction: In

    Pointer to the Fast I/O allocation to query.

  • size_out

    type: size_t *

    direction: Out

    Written with the size, in bytes, of the memory resource backing the allocation.

Return

Returns zero if successful; otherwise a non-zero value.

result nvidia_clara_fastio__size(
    IN nvidia_clara_fastio_alloc *alloc,
    OUT size_t *size_out
);

7.5.2.5. Map

Maps the memory resource of an nvidia_clara_fastio_alloc into the current process for read/write access. The pointer returned by this function remains valid until it is explicitly unmapped with nvidia_clara_fastio__unmap, or it is implicitly unmapped when the allocation is released with nvidia_clara_fastio__release or nvidia_clara_fastio__free.

Parameters

  • alloc

    type: nvidia_clara_fastio_alloc *

    direction: In

    Pointer to the Fast I/O allocation to map.

  • ptr_out

    type: void **

    direction: Out

    Pointer to the mapped memory resource.

Return

Returns zero if successful; otherwise a non-zero value.

result nvidia_clara_fastio__map(
    IN nvidia_clara_fastio_alloc *alloc,
    OUT void **ptr_out
);

7.5.2.6. Unmap

Unmaps a Fast I/O memory resource mapping previously returned by nvidia_clara_fastio__map.

Parameters

  • alloc

    type: nvidia_clara_fastio_alloc *

    direction: In

    Pointer to the Fast I/O allocation to unmap.

  • ptr

    type: void *

    direction: In

    Pointer to the memory mapping to unmap.

Return

Returns zero if successful; otherwise a non-zero value.

result nvidia_clara_fastio__unmap(
    IN nvidia_clara_fastio_alloc *alloc,
    IN void *ptr
);

7.5.3. Metadata Functions

The following functions expose the metadata state that is currently supported by a Fast I/O allocation. All of this metadata is provided for the convenience of Clara Applications, primarily for facilitating communication and data format negotiation between operators when publishing and retrieving allocations to and from a context. Clara does not currently use this metadata for any internal purposes.

7.5.3.1. Set Type And Dimensions

Sets the element type and dimensions of a Fast I/O allocation.

This metadata state allows allocations to be described as typed, N-dimensional arrays, which is typical of pipelines that pass tensor data between operators.

Setting the type and dimensions implies a required data size for the allocation, as follows. This function will fail if the allocation size does not match the required size.

requiredSize = elementSize * product(dimensions)

Parameters

  • alloc

    type: nvidia_clara_fastio_alloc *

    direction: In

    Pointer to the Fast I/O allocation to modify.

  • type

    type: nvidia_clara_element_type

    direction: In

    Element type of the allocation.

  • dimensions

    type: const size_t *

    direction: In

    Array of count values that provide the dimensions of the allocation.

  • count

    type: size_t

    direction: In

    Specifies the number of dimensions provided in the dimensions array.

Return

Returns zero if successful; otherwise a non-zero value.

result nvidia_clara_fastio__set_type_and_dimensions(
    IN nvidia_clara_fastio_alloc *alloc,
    IN nvidia_clara_element_type type,
    IN const size_t *dimensions,
    IN size_t count
);

7.5.3.2. Element Type

Returns the element type metadata of a Fast I/O allocation.

Set with nvidia_clara_fastio__set_type_and_dimensions; default value: NVIDIA_CLARA_ELEMENT_TYPE_BYTE.

Parameters

  • alloc

    type: nvidia_clara_fastio_alloc *

    direction: In

    Pointer to the Fast I/O allocation to query.

  • type_out

    type: nvidia_clara_element_type *

    direction: Out

    Written with the element type of the Fast I/O allocation.

Return

Returns zero if successful; otherwise a non-zero value.

result nvidia_clara_fastio__element_type(
    IN nvidia_clara_fastio_alloc *alloc,
    OUT nvidia_clara_element_type *type_out
);

7.5.3.3. Dimensions

Returns the number of dimensions in the metadata of a Fast I/O allocation.

Set with nvidia_clara_fastio__set_type_and_dimensions; default value: 1.

Parameters

  • alloc

    type: nvidia_clara_fastio_alloc *

    direction: In

    Pointer to the Fast I/O allocation to query.

  • dimensions_out

    type: size_t *

    direction: Out

    Written with the number of dimensions in the metadata of the Fast I/O allocation.

Return

Returns zero if successful; otherwise a non-zero value.

result nvidia_clara_fastio__dimensions(
    IN nvidia_clara_fastio_alloc *alloc,
    OUT size_t *dimensions_out
);

7.5.3.4. Dimension

Returns the size of a single dimension in the metadata of a Fast I/O allocation.

The value of axis must be in [0, dims-1], where dims is the number of dimensions in the allocation (see nvidia_clara_fastio__dimensions)

Set with nvidia_clara_fastio__set_type_and_dimensions; default is 1-dimension having the same size as the memory resource (in bytes).

Parameters

  • alloc

    type: nvidia_clara_fastio_alloc *

    direction: In

    Pointer to the Fast I/O allocation to query.

  • axis

    type: size_t

    direction: In

    The index of the dimension to query. Must be in [0, dims-1], where dims is the number of dimensions in the allocation (see nvidia_clara_fastio__dimensions)

  • dimension_out

    type: size_t *

    direction: Out

    Written with the size of the requested dimension of the allocation.

Return

Returns zero if successful; otherwise a non-zero value.

result nvidia_clara_fastio__dimension(
    IN nvidia_clara_fastio_alloc *alloc,
    IN size_t axis,
    OUT size_t *dimension_out
);

7.5.3.5. Num Elements

Returns the total number of elements in a Fast I/O allocation, where the total number of elements is defined as the product of the array dimensions.

Set with nvidia_clara_fastio__set_type_and_dimensions; default is equal to the number of bytes in the memory resource.

Parameters

  • alloc

    type: nvidia_clara_fastio_alloc *

    direction: In

    Pointer to the Fast I/O allocation to query.

  • num_elements_out

    type: size_t *

    direction: Out

    Written with the total number of elements based on the dimensions of the allocation.

Return

Returns zero if successful; otherwise a non-zero value.

result nvidia_clara_fastio__num_elements(
    IN nvidia_clara_fastio_alloc *alloc,
    OUT size_t *out
);