10. User Defined Pipelines

Using Clara Deploy SDK Pipeline Definition Language, you can build a custom pipeline. This section provides an overview of how to define a custom pipeline and how to execute it.

10.1. Build Custom Operators

A pipeline consists of one or more operators. If you are building a custom pipeline, you may already have access to custom Clara operators that you built earlier. If not, please refer to the section “Base Inference Application” to get information on how to build a Docker image to encapsulate your custom operator.

10.2. Place AI Model Files in the Right Directory

Copy all your AI models to the directory /clara/common/models/

10.3. Start With a Reference Pipeline

You can start with a reference pipeline, as you may find it easier to modify an existing pipeline to suit your needs.

  • Create a directory called “my-pipelines” in your development machine where you will store your pipelines.

  • Clara Reference Pipelines and associated artifacts are available at: /path-to-clara-deploy-install/sdk/clara-reference-app/Pipelines

  • Navigate to any of the sub-directories under the directory mentioned above and copy the YAML file to “my-pipelines” directory. The YAML file provides the definition of the reference pipeline. We will refer to your custom pipeline YAML file as “custom-pipeline.yaml”

10.4. Modify the Pipeline

A pipeline consists of multiple operators. Open “custom-pipeline.yaml” and add/modify/delete operators and associated parameters based on your needs. Pay close attention to the input and output parameters of each operator and ensure that they are chained properly to form a pipeline.

10.5. External DICOM Sender and DICOM Receiver

You need an external DICOM Service Class User (SCU) application to send images to the Clara DICOM Adapter (acting as a DICOM SCP). Similarly when your pipeline finishes executing, you may want to send the output to an external DICOM receiver. You may want to use an open-source DICOM toolkit called ‘dcmtk’ for external DICOM sender and DICOM receiver.

10.5.1. Install dcmtk

Install dcmtk utilities by issuing the following command:

sudo apt-get install dcmtk

10.5.2. Setup an External DICOM Receiver

You can use the storescp utility as an external DICOM receiver. Open a terminal and run the following command to setup a DICOM Receiver. Create a directory in your machine where DICOM images produced by the Clara Deploy SDK will be stored. We will refer to that directory as “dicom-destination-directory”.

mkdir path-to/dicom-destination-directory

rm -rf dicom-destination-directory/*

cd dicom-destination-directory

sudo storescp -v --fork -aet MYPACS 1004

MYPACS is the Application Entity (AE) title of the external DICOM Receiver.

10.5.3. Setup an External DICOM Sender

You can use the storescu utility from dcmmtk as a DICOM sender.

10.6. Publish the Pipeline

Publishing a pipeline registers it with the platform. Once a pipeline is registered, it is represented with an ID. Follow the steps below:

  • Open a terminal and navigate to the directory which contains the reference pipeline:

cd </path/to/my-pipelines>
  • Publish the pipeline by issuing the following command:

clara create pipelines -p my-pipeline.yaml

Clara CLI should successfully submit the pipeline and print a pipeline ID. Note down this pipeline ID.

  • Verify that the pipeline is correctly stored by issuing the following command:

kubectl get pipeline
  • In the list of pipeline IDs, you should see the pipeline ID created in the previous step.

10.7. Configure the Clara DICOM Destination

When you send images to the Clara Deploy SDK, the DICOM Adapter component acts as the internal DICOM Service Class Provider or a receiving agent.

  • Open the file ~/.clara/charts/dicom-adapter/files/dicom-server-config.yaml

  • In the SCP->sources section, ensure sender host-ip and ae-titles are correct. Alternatively, set reject-unknown-sources to false, which will allow any DICOM sender to create an association.

  • In the dicom->scp->ae-titles section, add a new ae title with the name “my-ae-title”.

  • In the pipeline-mappings section, define a new mapping with: “my-ae-title” as the name and clara-ae-title, and add the pipeline ID that was created before.

  • Update the IP address of the destination with the name MYPACS to the IP of your local machine.

  • Restart the DICOM server by issuing the following commands:

clara dicom stop <br>
clara dicom start
  • Verify that the DICOM Adapter is correctly started:

helm ls | grep dicom-adapter<br>
kubectl get pods | grep dicom-adapter

10.8. Trigger Pipeline

A pipeline is triggered when a DICOM series is sent to the specific AE title of the Clara DICOM Adapter which is associated with that pipeline. To trigger this pipeline you will need to have a set of relevant DICOM images that can be used as source input data.

  • Open a terminal and go to the directory which contains your DICOM images:

cd path-to/dicom-images
  • Trigger the pipeline (push the DICOM data to the proper ae title):

storescu -v +sd +r -xb -aet "DCM4CHEE" -aec "my-ae-title" AA.BBB.CC.DD 104 ./

Replace “AA.BBB.CC.DD” with the IP of the machine where the Clara Deploy SDK is deployed.

When the above step is successfully completed, you should see the following result in the terminal:

I: Received Store Response (Success) <br>
I: Releasing Association

10.9. Verify the Pipeline Execution in the Dashboard

  • Go to the Clara Dashboard UI using a web browser: The URL is: <IP of the machine>:8000

  • You should see a job with a name that includes the name of the pipeline you created.

  • You can inspect the status of each operator inside that job.

10.10. Verify the External DICOM Receiver Received the Images

If your pipeline outputs DICOM images and sends them to an external DICOM receiver, you may want to verify that the images were indeed received. If you used the storescp utility from dcmtk as mentioned earlier, please use a terminal to navigate to the “dicom-destination-directory” and verify whether the expected images are stored there. In addition, you may want to use a third party DICOM viewer to view those images.