8.11. Clara Pipeline Fast I/O Allocations
All Fast I/O memory resources provided by the Fast I/O Context are accessed by clara.FastIOAllocation
objects, which act as local references to the underlying memory resource.
In addition to pointing to the underlying Fast I/O memory resources, each clara.FastIOAllocaion
object also contain a set of metadata state which can be used to describe how the memory resources of that allocation are being used. For example, FastIOAllocation.set_type_and_dimensions
can be used to set the element type and array dimensions for a Fast I/O allocation being used to store typed, N-dimensional array data.
see also: Fast I/O Context
Every clara.FastIOAllocation
object returned by FastIOContext.create
or FastIOContext.get
acts as a reference to Fast I/O memory resources, which prevents the underlying memory resources from being deleted until all clara.FastIOAllocation
references to the same resource across all operators have been deleted.
The resource references held by clara.FastIOAllocation
objects will be released automatically when the object is deleted by the Python runtime. Alternatively, operators are provided more explicit control of releasing or freeing their resources using FastIOAllocation.release
or FastIOAllocation.free
. The difference between release
and free
is that while both immediately release the local reference to the underlying memory resource, free
will also mark the resource for deletion within the pipeline job, preventing any further access to the resources by any operator (though existing references and mappings will remain valid until they are also released). Freeing Fast I/O memory resources reduces the overall footprint of the pipeline job, and so allocations should be freed with FastIOAllocation.free
as soon as they are no longer needed by any operators in the pipeline.
However, since multiple operators in a pipeline may be consuming the same Fast I/O resource, it may not be known to the operator developer whether the Fast I/O inputs it consumes need to be freed after use or if they’ll be used again by another operator in the pipeline.
If the pipeline(s) using the operator are known to the operator developer, and they can guarantee which are the terminal operators for each allocation, then those terminal operators should also free those allocations.
If the pipeline(s) using the operator are unknown, or there is a possibility that additional operators may be added to a pipeline to consume the Fast I/O allocations, then the operator should let the underlying resource persist.
Note that all Fast I/O memory resources are limited to the pod that is executing the pipeline job, and so all resources will be implicitly freed once the pod has completed executing.
The following functions are considered the core Fast I/O allocation functions, and are responsible for managing the clara.FastIOAllocation
references in addition to providing functions to map the underlying memory resources for read and/or write access from the application.
8.11.2.1.Size
Returns the size, in bytes, of the memory resource backing the clara.FastIOAllocation
.
type: int
@property
def size(self)
8.11.2.2.Publish
Each Fast I/O allocation begins as a single clara.FastIOAllocation
object that is returned to the operator that created it using FastIOContext.create
. Fast I/O allocations are not accessible to other operators until FastIOAllocation.publish
is used to explicitly publish the allocation.
While clara.FastIOAllocation
objects act as local references to their underlying Fast I/O memory resources, they also store some basic metadata state that can be used to help describe the resources. For example, FastIOAllocation.set_type_and_dimensions
can be used to set the element type and array dimensions for Fast I/O allocations being used to store typed, N-dimensional arrays.
When multiple operators share access to a single Fast I/O memory resource, each operator will create its own clara.FastIOAllocation
object that will act as a reference to the underlying memory resource, as well as store its own metadata state to describe the resource. This means that the underlying memory resource is shared and may be accessed directly by multiple operators, but the metadata that each operator uses to describe the resources is copied into local state and not otherwise shared between operators.
Publishing a Fast I/O allocation with FastIOAllocation.publish
copies the allocation’s metadata and underlying memory reference to the context that created it using a unique string to identify the publication. After publication, the same string identifier can then be used by other operators executing in the pipeline to call FastIOContext.get
, which returns a clara.FastIOAllocation
that will reference the same underlying memory resource and has its metadata initialized with a copy of the state that was published.
Metadata published to the context is immutable, and any further updates to metadata in local allocation objects will not be reflected in previous context publications of that allocation. Rather than support mutable metadata, allocations can instead be published to the context any number or times, with potentially different metadata state each time, so long as each publication uses a different unique identifier.
The clara.FastIOAllocation
object remains valid after calling FastIOAllocation.publish
, and can still be used to access the memory resource and local metadata for that allocation.
see also: Fast I/O Context
Parameters
name
type:
string
direction: In
Name to use for the publication.
Return
No return value.
def publish(self, name)
8.11.2.3.Release
Releases a local clara.FastIOAllocation
reference. Releasing an allocation object will implicitly unmap any previously mapped pointers to the memory resource before releasing its reference to the underlying resource.
Since this function only releases the local allocation reference, the underlying Fast I/O memory resource that it pointed to will persist within the context, and thus can still be retrieved from a context using FastIOContext.get
(assuming it’s been previous published to the context). In order to both release the local reference and also free the underlying memory resources, preventing future use of the resource, use FastIOAllocation.free
.
Parameters
No parameters.
Return
No return value.
def release(self)
8.11.2.4.Free
Releases a local clara.FastIOAllocation
reference and frees the underlying Fast I/O memory resource. This behaves like FastIOAllocation.release
, except that the underlying Fast I/O memory resource is also marked to be freed, which prevents it from being accessed or mapped by any other operator. Note that existing mappings to the data in other operators will remain valid until unmapped, but attempting to map the resource after the free will fail.
Note that Fast I/O resources may be limited, and so freeing resources as early as possible within the pipeline is recommended to minimize resource usage. If possible, allocations should be freed by the last operator in the pipeline that accesses them.
Parameters
No parameters.
Return
No return value.
def free(self)
8.11.2.5.Map
Maps the memory resource of a clara.FastIOAllocation
into the current process for read/write access. The pointer returned by this function remains valid until it is explicitly unmapped with FastIOAllocation.unmap
, or it is implicitly unmapped when the clara.FastIOAllocation
object is released.
Parameters
No parameters.
Return
Pointer to the mapped Fast I/O memory resource.
type: (cdata 'void*')
def map(self)
8.11.2.5.1.Buffer
Returns an object of bytearray
type. The buffer is available only after calling map()
method. Otherwise, return None
type: bytearray
@property
def buffer(self)
8.11.2.6.Unmap
Unmaps a Fast I/O memory resource previously mapped by FastIOAllocation.map
.
Parameters
No parameters.
Return
No return value.
def unmap(self)
8.11.2.7.Map As Numpy Array
Helper functions that maps the Fast I/O memory resource, reads the type and dimensions in the allocation metadata, and returns a Numpy array that directly references the Fast I/O memory resource as its data storage.
The returned array is valid until the clara.FastIOAllocation
object is deleted or is explicitly unmapped with FastIOAllocation.unmap
.
The memory resources mapped into the Numpy array are mapped for read and write access, so the array contents can be modified in-place, but if any attempt is made to change the type or size of the Numpy array then the Numpy runtime will allocate a new data storage and break the mapping to the Fast I/O memory. See the Numpy documentation for more details on when arrays must be reallocated.
Parameters
No parameters.
Return
A Numpy array that uses the Fast I/O memory as its data storage.
type: numpy.array
def map_as_numpy_array(self)
The following functions expose the metadata state that is currently supported by a Fast I/O allocation. All of this metadata is provided for the convenience of Clara Applications, primarily for facilitating communication and data format negotiation between operators when publishing and retrieving allocations to and from a context. Clara only uses this metadata for utility function implementations, such as FastIOAllocation.map_as_numpy_array
.
8.11.3.1.Set Type And Dimensions
Sets the element type and dimensions of a Fast I/O allocation.
This metadata state allows allocations to be described as typed, N-dimensional arrays, which is typical of pipelines that pass tensor data between operators.
Setting the type and dimensions implies a required data size for the allocation, as follows. This function will fail if the allocation size does not match the required size.
requiredSize = elementSize * product(dimensions)
Parameters
element_type
type:
Clara or Numpy element type
direction: In
Element type of the allocation.
dimensions
type:
int list
direction: In
The dimensions for the allocation.
Return
No return value.
def set_type_and_dimensions(self, element_type, dimensions)
8.11.3.2.Element Type
Returns the element type metadata of a Fast I/O allocation.
Set with FastIOAllocation.set_type_and_dimensions
; default value: NVIDIA_CLARA_ELEMENT_TYPE_BYTE
.
type: NVIDIA_CLARA_ELEMENT_TYPE
@property
def element_type(self)
8.11.3.3.Dimensions
Returns the dimensions in the metadata of a Fast I/O allocation.
Set with FastIOAllocation.set_type_and_dimensions
; default is a single dimension having the same size as the memory resource (in bytes).
type: int list
@property
def dimensions()