7.11. Clara Pipeline Fast I/O Allocations

All Fast I/O memory resources provided by the Fast I/O Context are accessed by clara.FastIOAllocation objects, which act as local references to the underlying memory resource.

In addition to pointing to the underlying Fast I/O memory resources, each clara.FastIOAllocaion object also contain a set of metadata state which can be used to describe how the memory resources of that allocation are being used. For example, FastIOAllocation.set_type_and_dimensions can be used to set the element type and array dimensions for a Fast I/O allocation being used to store typed, N-dimensional array data.

see also: Fast I/O Context

7.11.1. Allocation Ownership and Lifespan

Every clara.FastIOAllocation object returned by FastIOContext.create or FastIOContext.get acts as a reference to Fast I/O memory resources, which prevents the underlying memory resources from being deleted until all clara.FastIOAllocation references to the same resource across all operators have been deleted.

The resource references held by clara.FastIOAllocation objects will be released automatically when the object is deleted by the Python runtime. Alternatively, operators are provided more explicit control of releasing or freeing their resources using FastIOAllocation.release or FastIOAllocation.free. The difference between release and free is that while both immediately release the local reference to the underlying memory resource, free will also mark the resource for deletion within the pipeline job, preventing any further access to the resources by any operator (though existing references and mappings will remain valid until they are also released). Freeing Fast I/O memory resources reduces the overall footprint of the pipeline job, and so allocations should be freed with FastIOAllocation.free as soon as they are no longer needed by any operators in the pipeline.

However, since multiple operators in a pipeline may be consuming the same Fast I/O resource, it may not be known to the operator developer whether the Fast I/O inputs it consumes need to be freed after use or if they’ll be used again by another operator in the pipeline.

If the pipeline(s) using the operator are known to the operator developer, and they can guarantee which are the terminal operators for each allocation, then those terminal operators should also free those allocations.

If the pipeline(s) using the operator are unknown, or there is a possibility that additional operators may be added to a pipeline to consume the Fast I/O allocations, then the operator should let the underlying resource persist.

Note that all Fast I/O memory resources are limited to the pod that is executing the pipeline job, and so all resources will be implicitly freed once the pod has completed executing.

7.11.2. Core Allocation and Memory Management Functions

The following functions are considered the core Fast I/O allocation functions, and are responsible for managing the clara.FastIOAllocation references in addition to providing functions to map the underlying memory resources for read and/or write access from the application.

7.11.2.1. Size

Returns the size, in bytes, of the memory resource backing the clara.FastIOAllocation.

type: int

@property
def size(self)

7.11.2.2. Publish

Each Fast I/O allocation begins as a single clara.FastIOAllocation object that is returned to the operator that created it using FastIOContext.create. Fast I/O allocations are not accessible to other operators until FastIOAllocation.publish is used to explicitly publish the allocation.

While clara.FastIOAllocation objects act as local references to their underlying Fast I/O memory resources, they also store some basic metadata state that can be used to help describe the resources. For example, FastIOAllocation.set_type_and_dimensions can be used to set the element type and array dimensions for Fast I/O allocations being used to store typed, N-dimensional arrays.

When multiple operators share access to a single Fast I/O memory resource, each operator will create its own clara.FastIOAllocation object that will act as a reference to the underlying memory resource, as well as store its own metadata state to describe the resource. This means that the underlying memory resource is shared and may be accessed directly by multiple operators, but the metadata that each operator uses to describe the resources is copied into local state and not otherwise shared between operators.

Publishing a Fast I/O allocation with FastIOAllocation.publish copies the allocation’s metadata and underlying memory reference to the context that created it using a unique string to identify the publication. After publication, the same string identifier can then be used by other operators executing in the pipeline to call FastIOContext.get, which returns a clara.FastIOAllocation that will reference the same underlying memory resource and has its metadata initialized with a copy of the state that was published.

Metadata published to the context is immutable, and any further updates to metadata in local allocation objects will not be reflected in previous context publications of that allocation. Rather than support mutable metadata, allocations can instead be published to the context any number or times, with potentially different metadata state each time, so long as each publication uses a different unique identifier.

The clara.FastIOAllocation object remains valid after calling FastIOAllocation.publish, and can still be used to access the memory resource and local metadata for that allocation.

see also: Fast I/O Context

Parameters

  • name

    type: string

    direction: In

    Name to use for the publication.

Return

No return value.

def publish(self, name)

7.11.2.3. Release

Releases a local clara.FastIOAllocation reference. Releasing an allocation object will implicitly unmap any previously mapped pointers to the memory resource before releasing its reference to the underlying resource.

Since this function only releases the local allocation reference, the underlying Fast I/O memory resource that it pointed to will persist within the context, and thus can still be retrieved from a context using FastIOContext.get (assuming it’s been previous published to the context). In order to both release the local reference and also free the underlying memory resources, preventing future use of the resource, use FastIOAllocation.free.

Parameters

No parameters.

Return

No return value.

def release(self)

7.11.2.4. Free

Releases a local clara.FastIOAllocation reference and frees the underlying Fast I/O memory resource. This behaves like FastIOAllocation.release, except that the underlying Fast I/O memory resource is also marked to be freed, which prevents it from being accessed or mapped by any other operator. Note that existing mappings to the data in other operators will remain valid until unmapped, but attempting to map the resource after the free will fail.

Note that Fast I/O resources may be limited, and so freeing resources as early as possible within the pipeline is recommended to minimize resource usage. If possible, allocations should be freed by the last operator in the pipeline that accesses them.

Parameters

No parameters.

Return

No return value.

def free(self)

7.11.2.5. Map

Maps the memory resource of a clara.FastIOAllocation into the current process for read/write access. The pointer returned by this function remains valid until it is explicitly unmapped with FastIOAllocation.unmap, or it is implicitly unmapped when the clara.FastIOAllocation object is released.

Parameters

No parameters.

Return

Pointer to the mapped Fast I/O memory resource.

type: (cdata 'void*')

def map(self)

7.11.2.5.1. Buffer

Returns an object of bytearray type. The buffer is available only after calling map() method. Otherwise, return None

type: bytearray

@property
def buffer(self)

7.11.2.6. Unmap

Unmaps a Fast I/O memory resource previously mapped by FastIOAllocation.map.

Parameters

No parameters.

Return

No return value.

def unmap(self)

7.11.2.7. Map As Numpy Array

Helper functions that maps the Fast I/O memory resource, reads the type and dimensions in the allocation metadata, and returns a Numpy array that directly references the Fast I/O memory resource as its data storage.

The returned array is valid until the clara.FastIOAllocation object is deleted or is explicitly unmapped with FastIOAllocation.unmap.

The memory resources mapped into the Numpy array are mapped for read and write access, so the array contents can be modified in-place, but if any attempt is made to change the type or size of the Numpy array then the Numpy runtime will allocate a new data storage and break the mapping to the Fast I/O memory. See the Numpy documentation for more details on when arrays must be reallocated.

Parameters

No parameters.

Return

A Numpy array that uses the Fast I/O memory as its data storage.

type: numpy.array

def map_as_numpy_array(self)

7.11.3. Metadata Functions

The following functions expose the metadata state that is currently supported by a Fast I/O allocation. All of this metadata is provided for the convenience of Clara Applications, primarily for facilitating communication and data format negotiation between operators when publishing and retrieving allocations to and from a context. Clara only uses this metadata for utility function implementations, such as FastIOAllocation.map_as_numpy_array.

7.11.3.1. Set Type And Dimensions

Sets the element type and dimensions of a Fast I/O allocation.

This metadata state allows allocations to be described as typed, N-dimensional arrays, which is typical of pipelines that pass tensor data between operators.

Setting the type and dimensions implies a required data size for the allocation, as follows. This function will fail if the allocation size does not match the required size.

requiredSize = elementSize * product(dimensions)

Parameters

  • element_type

    type: Clara or Numpy element type

    direction: In

    Element type of the allocation.

  • dimensions

    type: int list

    direction: In

    The dimensions for the allocation.

Return

No return value.

def set_type_and_dimensions(self, element_type, dimensions)

7.11.3.2. Element Type

Returns the element type metadata of a Fast I/O allocation.

Set with FastIOAllocation.set_type_and_dimensions; default value: NVIDIA_CLARA_ELEMENT_TYPE_BYTE.

type: NVIDIA_CLARA_ELEMENT_TYPE

@property
def element_type(self)

7.11.3.3. Dimensions

Returns the dimensions in the metadata of a Fast I/O allocation.

Set with FastIOAllocation.set_type_and_dimensions; default is a single dimension having the same size as the memory resource (in bytes).

type: int list

@property
def dimensions()