13. User Defined Pipelines
Using Clara Deploy SDK Pipeline Definition Language, you can build a custom pipeline. This section provides an overview of how to define a custom pipeline and how to execute it.
A pipeline consists of one or more operators. If you are building a custom pipeline, you may already have access to custom Clara operators that you built earlier. If not, please refer to the section “Base Inference Application” to get information on how to build a Docker image to encapsulate your custom operator.
Copy all your AI models to the directory /clara/common/models/
You can start with a reference pipeline, as you may find it easier to modify an existing pipeline to suit your needs.
Create a directory called “my-pipelines” in your development machine where you will store your pipelines.
Clara Reference Pipelines and associated artifacts are available at:
/path-to-clara-deploy-install/sdk/clara-reference-app/Pipelines
Navigate to any of the sub-directories under the directory mentioned above and copy the YAML file to “my-pipelines” directory. The YAML file provides the definition of the reference pipeline. We will refer to your custom pipeline YAML file as “custom-pipeline.yaml”
A pipeline consists of multiple operators. Open “custom-pipeline.yaml” and add/modify/delete operators and associated parameters based on your needs. Pay close attention to the input and output parameters of each operator and ensure that they are chained properly to form a pipeline.
You need an external DICOM Service Class User (SCU) application to send images to the Clara DICOM Adapter (acting as a DICOM SCP). Similarly when your pipeline finishes executing, you may want to send the output to an external DICOM receiver. You may want to use an open-source DICOM toolkit called ‘dcmtk’ for external DICOM sender and DICOM receiver.
13.5.1.Install dcmtk
Install dcmtk utilities by issuing the following command:
sudo apt-get install dcmtk
13.5.2.Setup an External DICOM Receiver
You can use the storescp utility as an external DICOM receiver. Open a terminal and run the following command to setup a DICOM Receiver. Create a directory in your machine where DICOM images produced by the Clara Deploy SDK will be stored. We will refer to that directory as “dicom-destination-directory”.
mkdir path-to/dicom-destination-directory
rm -rf dicom-destination-directory/*
cd dicom-destination-directory
sudo storescp -v --fork -aet MYPACS 1004
MYPACS is the Application Entity (AE) title of the external DICOM Receiver.
13.5.3.Setup an External DICOM Sender
You can use the storescu utility from dcmmtk as a DICOM sender.
Creating a pipeline registers it with the platform. Once a pipeline is registered, it is represented with an ID. Follow the steps below:
Open a terminal and navigate to the directory <my-pipelines></my-pipelines> which contains the reference pipeline:
cd </path/to/my-pipelines>
Create the pipeline by issuing the following command:
clara create pipeline -p my-pipeline.yaml
Clara CLI should successfully submit the pipeline and print a pipeline ID. Note down this pipeline ID.
Verify that the pipeline is correctly stored by issuing the following command:
kubectl get pipeline
In the list of pipeline IDs, you should see the pipeline ID created in the previous step.
When you send images to the Clara Deploy SDK, the DICOM Adapter component acts as the internal DICOM Service Class Provider or a receiving agent.
Open the file
~/.clara/charts/dicom-adapter/files/dicom-server-config.yaml
In the SCP->sources section, ensure sender host-ip and ae-titles are correct. Alternatively, set reject-unknown-sources to false, which will allow any DICOM sender to create an association.
In the dicom->scp->ae-titles section, add a new ae title with the name “my-ae-title”.
In the pipeline-mappings section, define a new mapping with: “my-ae-title” as the name and clara-ae-title, and add the pipeline ID that was created before.
Update the IP address of the destination with the name MYPACS to the IP of your local machine.
Restart the DICOM server by issuing the following commands:
clara dicom restart
Verify that the DICOM Adapter is correctly started:
helm ls | grep dicom-adapter<br>
kubectl get pods | grep dicom-adapter
A pipeline is triggered when a DICOM series is sent to the specific AE title of the Clara DICOM Adapter which is associated with that pipeline. To trigger this pipeline you will need to have a set of relevant DICOM images that can be used as source input data.
Open a terminal and go to the directory which contains your DICOM images:
cd path-to/dicom-images
Trigger the pipeline (push the DICOM data to the proper ae title):
storescu -v +sd +r -xb -aet "DCM4CHEE" -aec "my-ae-title" AA.BBB.CC.DD 104 ./
Replace “AA.BBB.CC.DD” with the IP of the machine where the Clara Deploy SDK is deployed.
When the above step is completed, you should see the following result in the terminal:
I: Received Store Response (Success) <br>
I: Releasing Association
Go to the Clara Dashboard UI using a web browser: The URL is:
<IP of the machine>
:8000You should see a job with a name that includes the name of the pipeline you created.
You can inspect the status of each operator inside that job.
If your pipeline outputs DICOM images and sends them to an external DICOM receiver, you may want to verify that the images were indeed received. If you used the storescp utility from dcmtk as mentioned earlier, please use a terminal to navigate to the “dicom-destination-directory” and verify whether the expected images are stored there. Also, you may want to use a third party DICOM viewer to view those images.