7.12. Clara Pipeline Fast I/O Context¶
clara.Payload traditionally provides disk-based input and output support to applications, it is often the case that medical imaging pipelines will be operating on very large data sets, sharing and/or passing data between the operators in the pipeline, and swapping this data to and from disk between each operator is inefficient in both runtime and resource usage.
To avoid the inefficient use of disk-based data passing, the
clara.Payload also provides a Fast I/O Context,
clara.FastIOContext, which is an object that provides access to memory resources that are accessible by all operators running within the same pipeline, and which can be used for for efficient, zero-copy sharing and passing of data between operators.
The Fast I/O Context provides not just the means to allocate Fast I/O memory resources (
FastIOContext.create), but also provides a mechanism to pass these resources between multiple operators in a pipeline using basic string identifiers (e.g. using
clara.FastIOContext object can be retrieved from the
Payload.fastio_context property within the
clara.Payload object that was provided to the application as part of the Clara Pipeline Operator Life-Cycle execution event callback. This object is only valid throughout the lifetime of the application supplied execution callback handler.
Allocates a Fast I/O memory resource of the requested size. Returns a
clara.FastIOAllocation object to identify the allocation.
Fast I/O allocations returned by this function are initially only accessible by the calling process. In order to make a Fast I/O allocation available to other operators in the pipeline, the allocating process must publish the allocation to the Fast I/O context using
FastIOAllocation.publish, after which other operators may access and import a local reference to that same resource using
see also: Fast I/O Allocations
Size, in bytes, of the requested allocation.
clara.FastIOAllocation object for the new allocation.
def create(self, size)
Allocations that were previously published to a Fast I/O context using
FastIOAllocation.publish can be retrieved by any other operator in the same pipeline job by calling
FastIOContext.get using the same identifier that was used to publish it. This will return a new
clara.FastIOAllocation object that references the same underlying memory resource and has its metadata initialized with a copy of the metadata that was published.
The allocations returned by this function act identically to those returned by
FastIOContext.create, and do not have any restrictions on their use. The metadata of these objects can be modified, the allocations can be published to the context, and they can be used to free the underlying memory resources with
see also: Fast I/O Allocations
Name of the publication to retrieve.
clara.FastIOAllocation object for the retrieved allocation.
def get(self, name)
7.12.3. Publish Array¶
This utility function publishes Numpy arrays directly to a Fast I/O context without needing to manually create, initialize, and publish a
This function does the following:
Allocates a Fast I/O memory region large enough for the provided numpy array.
Sets the metadata (type/dimensions) of the Fast I/O allocation to match that of the array.
Copies the contents of the array to the Fast I/O memory.
Publishes the allocation to the context using the given idendifier.
Numpy array to publish.
Name to use for the array’s Fast I/O publication.
No return value.
def publish_array(self, array, name)