12.17. FastIO Passthrough Pipeline Using FastIO Variable API

This is a simple passthrough pipeline that uses the typed-operator API (available as of Pipeline API Version 0.5.0).

There are three main features in this pipeline:

• Operator input and output type-definitions must exist and must match each other’s types if connected. This ensures that if operators are swapped in and out they must:
  • match the pipeline’s type definition, and that
  • one operators output will match another’s input.
• The type of the FastIO variable is available inside the operator, so any input to it can be converted to the correct format before assignment.

The FastIO variable feature is only available for Pipeline api-versions 0.5.0 and above, and only through the Clara Pipeline Driver (orchestrator: Clara). For more details on the Pipeline definition see the next section.

12.17.1. Pipeline Definition

There are three operators:

• nifti-to-fastio reads the input payload tot he pipeline, and only accepts “Nifti” files. It outputs two FastIO variables:
  • segmentation - an array of float32 elements of size [1, -1, -1, -1] where the -1 elements in the shape indicate a size that can be changed by the operator at runtime (in this example this is a one-channel 3D image of the size of the array contained in the Nifti file);
  • segmentation_shape a fixed-size array of int32 elements of size [4] whose size cannot be changed at runtime (but of course values can be updated). This operator updates the entries of segmentation_shape to match the shape of the segmentation array so that we can verify in th enext operator that these match.
• fastio-to-npz reads both of the above outputs and performs a routine check that the shape of the array in segmentation matches segmentation_shape. This is not neccesary, however, here it is used to highlight the dynamic and fixed shapes one can specify in the input and output type defintions to constrain how operators are replaced in the future. This operators output the FastIO variable as a compressed Numpy file.
• compare-original-to-output performs a routine check that the input payload array and the output of fastio-to-npz are in fact identical.

            

            
api-version: 0.5.0
orchestrator: Clara
name: fastio-passthrough-dynamic-pipeline
operators:
- name: nifti-to-fastio
  description: Take Nifti input and transfer to FastIO array.
  container:
    image: clara/nifti-to-fastio
    tag: latest
  input:
  - path: /input
    type: stream
    element-type: nifti
  output:
  - name: segmentation
    type: array
    element-type: float32
    shape: [1, -1, -1, -1]    # 4 dims = [1 channel, any size, any size, any size]
  - name: segmentation_shape  # this output contains the actual shape and batch size
    type: array
    element-type: int32
    shape: [4]                 # 4 dims = [# channels, dim x, dim y, dim z]
- name: fastio-to-npz
  description: Take FastIO array and output as numpy array file (npz).
  container:
    image: clara/fastio-to-npz
    tag: latest
  input:
  - from: nifti-to-fastio
    name: segmentation
    type: array
    element-type: float32
    shape: [1, -1, -1, -1]
  - from: nifti-to-fastio
    name: segmentation_shape
    type: array
    element-type: int32
    shape: [4]
  output:
  - name: numpy_npz
    path: /output
    type: stream
    element-type: npz
- name: compare-original-to-output
  description: Compare the output of the NPZ file to the original Nifti
  container:
    image: clara/compare-input-to-npz
    tag: latest
  input:
    - path: /input
      type: stream
      element-type: nifti
    - from: fastio-to-npz
      name: numpy_npz
      path: /npz
      type: stream
      element-type: npz
  output:
    - name: truth-val
      path: /output
      type: stream
      element-type: txt
        

12.17.2. Data Input

Input requires a folder containing a Nifti file (.nii or .nii.gz, if more than one exists only the first one found will be used), or a single Nifti file.

12.17.3. Data Output

A text (.txt) file with true or false in it depending on whether the original array and the output array are equal.

12.17.4. Executing the Pipeline

A pipeline can be triggered via Clara CLI. To trigger this pipeline, you will need to have a volume image in Nifti format (.nii/.nii.gz) that can be used as source input data.

1. Open a terminal and go to the pipeline directory which has pipeline definition file as well as test input/model folder.

            

            
cd /path-to-pipeline-folder/
        

1. Trigger the pipeline (upload input files to Clara API Server) with the pipeline ID in the previous section.

            

            
PIPELINE_ID=clara list pipelines | grep fastio-passthrough-dynamic-pipeline | awk '{print $2}'
clara create jobs -n fastio-variable-test -p $PIPELINE_ID -f <TEST_DATA_FOLDER>/<FILE-NAME>(.nii|.nii.gz)
        

When the above step is successfully completed, you should be able to see the following result in the terminal (UUID of Job and Payload will be different). E.g.

            

            
JOB_ID: daa5aea74d1446a8a24b9d3990bfef67
PAYLOAD_ID: 79aa5500e6134cf888f1d6d89f2e249e
        

Note down this job ID and payload ID as they would be used in a later section to access output files.

Then, start the job:

            

            
# Start a job
clara start job -j <JOB ID>
        

12.17.5. Check Job Status and Download/View Payloads in Clara Console/Dashboard

With Clara Orchestrator (API version >= 0.4.0)

• Go to the Clara Management Console using a web browser: The URL is: <IP of the machine>:32002
• Go to the JOBS view. You should see a job with a name that includes the name of the job you specified
• You can inspect the status of each operator inside that job

Please check Clara Management Console to see the web application in detail.

With Argo Orchestrator (API version <= 0.3.0)

Go to the Clara dashboard UI using a web browser: The URL is <machine-ip>:8000.

• You should see a job with a name that includes the name of the pipeline you created.
• You can inspect the status of each operator inside that job

12.17.6. Download and Verify the Outputs

Once the job completes successfully, we can download the output by using clara download command.

Output payload files are located under /operators/<operator name defined in the pipeline definition&gt;/ folder. Since we triggered a pipeline whose volume output operator’s name is compare-input-to-npz, we can use the following command to download output files into result folder:

            

            
clara download <Job ID>:/operators/compare-input-to-npz/* result
        

Then, you can see the output files using the following command:

            

            
cat result/truth-val/comparison.txt && echo ""
        

If you are working in the machine where Clara was deployed, the payload folders are directly accessible at the default payload folder /clara/payloads/<Payload ID>/.

Release Notes, the Getting Started Guide, and the SDK itself are available at the NVIDIA Developer forum: (https://developer.nvidia.com/clara).

For answers to any questions you may have about this release, visit the NVIDIA Devtalk forum: (https://devtalk.nvidia.com/default/board/362/clara-sdk/).