12.17. FastIO Passthrough Pipeline Using FastIO Variable API
This is a simple passthrough pipeline that uses the typed-operator API (available as of Pipeline API Version 0.5.0).
There are three main features in this pipeline:
Operator input and output type-definitions must exist and must match each other’s types if connected. This ensures that if operators are swapped in and out they must:
match the pipeline’s type definition, and that
one operators output will match another’s input.
The type of the FastIO variable is available inside the operator, so any input to it can be converted to the correct format before assignment.
The FastIO variable feature is only available for Pipeline api-version
s 0.5.0
and above, and only through the Clara Pipeline Driver (orchestrator: Clara
). For more details on the Pipeline definition see the next section.
12.17.1.Pipeline Definition
There are three operators:
nifti-to-fastio
reads the input payload tot he pipeline, and only accepts “Nifti” files. It outputs two FastIO variables:segmentation
- anarray
offloat32
elements of size[1, -1, -1, -1]
where the-1
elements in theshape
indicate a size that can be changed by the operator at runtime (in this example this is a one-channel 3D image of the size of the array contained in the Nifti file);segmentation_shape
a fixed-sizearray
ofint32
elements of size[4]
whose size cannot be changed at runtime (but of course values can be updated). This operator updates the entries ofsegmentation_shape
to match theshape
of thesegmentation
array so that we can verify in th enext operator that these match.
fastio-to-npz
reads both of the above outputs and performs a routine check that theshape
of the array insegmentation
matchessegmentation_shape
. This is not neccesary, however, here it is used to highlight the dynamic and fixedshape
s one can specify in the input and output type defintions to constrain how operators are replaced in the future. This operators output the FastIO variable as a compressed Numpy file.compare-original-to-output
performs a routine check that the input payload array and the output offastio-to-npz
are in fact identical.
api-version: 0.5.0
orchestrator: Clara
name: fastio-passthrough-dynamic-pipeline
operators:
- name: nifti-to-fastio
description: Take Nifti input and transfer to FastIO array.
container:
image: clara/nifti-to-fastio
tag: latest
input:
- path: /input
type: stream
element-type: nifti
output:
- name: segmentation
type: array
element-type: float32
shape: [1, -1, -1, -1] # 4 dims = [1 channel, any size, any size, any size]
- name: segmentation_shape # this output contains the actual shape and batch size
type: array
element-type: int32
shape: [4] # 4 dims = [# channels, dim x, dim y, dim z]
- name: fastio-to-npz
description: Take FastIO array and output as numpy array file (npz).
container:
image: clara/fastio-to-npz
tag: latest
input:
- from: nifti-to-fastio
name: segmentation
type: array
element-type: float32
shape: [1, -1, -1, -1]
- from: nifti-to-fastio
name: segmentation_shape
type: array
element-type: int32
shape: [4]
output:
- name: numpy_npz
path: /output
type: stream
element-type: npz
- name: compare-original-to-output
description: Compare the output of the NPZ file to the original Nifti
container:
image: clara/compare-input-to-npz
tag: latest
input:
- path: /input
type: stream
element-type: nifti
- from: fastio-to-npz
name: numpy_npz
path: /npz
type: stream
element-type: npz
output:
- name: truth-val
path: /output
type: stream
element-type: txt
12.17.2.Data Input
Input requires a folder containing a Nifti file (.nii
or .nii.gz
, if more than one exists only the first one found will be used), or a single Nifti file.
12.17.3.Data Output
A text (.txt
) file with true
or false
in it depending on whether the original array and the output array are equal.
12.17.4.Executing the Pipeline
A pipeline can be triggered via Clara CLI. To trigger this pipeline, you will need to have a volume image in Nifti format (.nii
/.nii.gz
) that can be used as source input data.
Open a terminal and go to the pipeline directory which has pipeline definition file as well as test input/model folder.
cd /path-to-pipeline-folder/
Trigger the pipeline (upload input files to Clara API Server) with the pipeline ID in the previous section.
PIPELINE_ID=clara list pipelines | grep fastio-passthrough-dynamic-pipeline | awk '{print $2}'
clara create jobs -n fastio-variable-test -p $PIPELINE_ID -f <TEST_DATA_FOLDER>/<FILE-NAME>(.nii|.nii.gz)
When the above step is successfully completed, you should be able to see the following result in the terminal (UUID of Job and Payload will be different). E.g.
JOB_ID: daa5aea74d1446a8a24b9d3990bfef67
PAYLOAD_ID: 79aa5500e6134cf888f1d6d89f2e249e
Note down this job ID and payload ID as they would be used in a later section to access output files.
Then, start the job:
# Start a job
clara start job -j <JOB ID>
12.17.5.Check Job Status and Download/View Payloads in Clara Console/Dashboard
With Clara Orchestrator (API version >= 0.4.0)
Go to the Clara Management Console using a web browser: The URL is:
<IP of the machine>:32002
Go to the
JOBS
view. You should see a job with a name that includes the name of the job you specifiedYou can inspect the status of each operator inside that job
Please check Clara Management Console to see the web application in detail.
With Argo Orchestrator (API version <= 0.3.0)
Go to the Clara dashboard UI using a web browser: The URL is <machine-ip>:8000
.
You should see a job with a name that includes the name of the pipeline you created.
You can inspect the status of each operator inside that job
12.17.6.Download and Verify the Outputs
Once the job completes successfully, we can download the output by using clara download command.
Output payload files are located under /operators/<operator name defined in the pipeline definition>/ folder.
Since we triggered a pipeline whose volume output operator’s name is compare-input-to-npz
, we can use the following command to
download output files into result
folder:
clara download <Job ID>:/operators/compare-input-to-npz/* result
Then, you can see the output files using the following command:
cat result/truth-val/comparison.txt && echo ""
If you are working in the machine where Clara was deployed, the payload folders are directly accessible at the default payload folder /clara/payloads/<Payload ID>/
.
Release Notes, the Getting Started Guide, and the SDK itself are available at the NVIDIA Developer forum: (https://developer.nvidia.com/clara).
For answers to any questions you may have about this release, visit the NVIDIA Devtalk forum: (https://devtalk.nvidia.com/default/board/362/clara-sdk/).