Clara Pipeline Generation Guide (Internal document, not updated)
Execute the Clara reference pipeline locally with the following command:
cd `git rev-parse --show-toplevel`/deploy/clara-reference-workflow
./clara-wf test # default: ai-only test
Execute other pipelines locally with the following commands:
./clara-wf test 0fcdcdea-89c6-456f-b20a-bc7379c54ef2 # ct-ai, not verified yet
./clara-wf test 1db65f99-c9b7-4329-ab9c-d519e0557638 # organ-seg
./clara-wf test fd3ee8bf-b9f3-4808-bd60-243f870ff9bd # liver-seg, not verified yet
./clara-wf test a78b1e7a-0325-4d42-b90f-207eb576b14f # ct-partial
./clara-wf test 1995f10e-ee14-4d67-b307-452051637dbb # dicom-reader-only
./clara-wf test 8371285c-8ea8-41cd-958e-44e5664c6ec3 # dicom-reader-writer
./clara-wf test 613e3fce-0d96-495a-a760-f43e68deefd8 # recon-only
./clara-wf test 21dd9b92-f307-4e68-ba31-fe8b1449ceea # recon-ai
./clara-wf test 50d31ed4-eeac-47e6-b584-c562f22a0091 # ai-only
./clara-wf test eb66f705-13ea-402c-8baa-143fb3fd9cd5 # dicom-writer-only
The first time you execute a pipeline you need the nvcr.io API key. Obtain that at the following website: https://ngc.nvidia.com/configuration/api-key
Set NGC_API_KEY environment variable if you want to set the API key without user interaction.
gbae@gigony:/ssd/gitlab/clara/devenv/sdk/deploy/clara-reference-workflow$ ./clara-wf test
Error from server (NotFound): secrets "nvcr.io" not found
Get NGC API key from https://ngc.nvidia.com/configuration/api-key
Then, enter the NGC API key:
secret/nvcr.io created
TRTIS is not deployed. Installing TRTIS...
NAME: trtis
LAST DEPLOYED: Tue Feb 19 10:45:54 2019
NAMESPACE: default
STATUS: DEPLOYED
RESOURCES:
==> v1/Service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
trtis ClusterIP 10.99.58.169 <none> 8000/TCP,8001/TCP,8002/TCP 12s
==> v1/Deployment
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
trtis 1 1 1 1 12s
==> v1/Pod(related)
NAME READY STATUS RESTARTS AGE
trtis-5586fb977f-b5swh 1/1 Running 0 12s
NOTES:
Get the application URL with the following commands:
export POD_NAME=$(kubectl get pods --namespace default -l "app.kubernetes.io/name=trtis,app.kubernetes.io/instance=trtis"-o jsonpath="{.items[0].metadata.name}")
echo "Visit http://127.0.0.1:8080 to use your application"
kubectl port-forward $POD_NAME 8080:8000
Output is similar to the following:
TRTIS Pod Name : trtis-5586fb977f-b5swh
NVIDIA_CLARA_TRTISURI : 10.99.58.169:8000
Sending build context to Docker daemon 446.2MB
Step 1/20 : FROM ubuntu:16.04
---> 7e87e2b3bf7a
Step 2/20 : ENV PYVER=3.5
---> Using cache
---> 9450c12cd9d9
Step 3/20 : ENV LC_ALL=C.UTF-8
---> Using cache
---> 544d9f54d099
Step 4/20 : ENV LANG=C.UTF-8
---> Using cache
---> 0fba046f8f9c
Step 5/20 : ENV USER=root
---> Using cache
---> e2bc9b37a014
Step 6/20 : ENV HOME=/root
---> Using cache
---> ef5a9e22dfa1
Step 7/20 : RUN apt-get update && apt-get install -y --no-install-recommends python3.5 curl libcurl3 rsync ca-certificates && curl -O https://bootstrap.pypa.io/get-pip.py && rm -rf /var/lib/apt/lists/*
---> Using cache
---> e63e95fd3adf
Step 8/20 : RUN rm -f /usr/bin/python && rm -f /usr/bin/python`echo $PYVER | cut -c1-1` && ln -s /usr/bin/python$PYVER /usr/bin/python && ln -s /usr/bin/python$PYVER /usr/bin/python`echo $PYVER | cut -c1-1`
---> Using cache
---> 4b43cbdb82a1
Step 9/20 : RUN python$PYVER get-pip.py && rm get-pip.py
---> Using cache
---> 81d740a157b0
Step 10/20 : WORKDIR /app
---> Using cache
---> fde61fa86219
Step 11/20 : RUN mkdir -p input output
---> Using cache
---> 92dd23ee1f7a
Step 12/20 : COPY ./sdk_dist/*.whl ./sdk_dist/
---> Using cache
---> af363d6802b2
Step 13/20 : COPY ./trtis_client/*.whl ./trtis_client/
---> Using cache
---> fce8222e35d2
Step 14/20 : COPY ./app_vnet ./app_vnet
---> Using cache
---> a022c4ffd9f5
Step 15/20 : COPY ./Pipfile ./
---> Using cache
---> 198e66a82e3c
Step 16/20 : RUN pip install --upgrade setuptools pipenv
---> Using cache
---> 56e71ad5dec4
Step 17/20 : RUN pipenv install --python 3.5 ./trtis_client/*.whl ./sdk_dist/*.whl --skip-lock && pipenv run pip install -r ./app_vnet/requirements.txt
---> Using cache
---> 4c327317cc6f
Step 18/20 : RUN rm -rf $(pipenv --venv)/lib/python3.5/site-packages/future/backports/test/*.pem
---> Using cache
---> 9bb79383e05e
Step 19/20 : RUN chmod -R 777 /root
---> Using cache
---> c665700a8b38
Step 20/20 : ENTRYPOINT ["pipenv", "run", "python", "app_vnet/main.py"]
---> Using cache
---> a52298d0027d
Successfully built a52298d0027d
Successfully tagged ai-vnet:latest
NAME: test-wf
LAST DEPLOYED: Mon Feb 25 00:34:58 2019
NAMESPACE: default
STATUS: DEPLOYED
RESOURCES:
==> v1/PersistentVolumeClaim
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
pv-clara-payload-volume-claim-00000000-0000-0000-0000-000000000000 Bound pv-clara-payload-volume-00000000-0000-0000-0000-000000000000 20Gi RWO pv-clara-payload-00000000-0000-0000-0000-000000000000 2s
==> v1/Job
NAME COMPLETIONS DURATION AGE
test-wf-clara-workflow 0/1 2s 2s
==> v1/Pod(related)
NAME READY STATUS RESTARTS AGE
test-wf-clara-workflow-z5tds 0/1 ContainerCreating 0 2s
==> v1/PersistentVolume
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pv-clara-payload-volume-00000000-0000-0000-0000-000000000000 20Gi RWO Retain Bound default/pv-clara-payload-volume-claim-00000000-0000-0000-0000-000000000000 pv-clara-payload-00000000-0000-0000-0000-000000000000 2s
NOTES:
Job 'Test Job' (00000000-0000-0000-0000-000000000000) executed!
release "test-wf" deleted
release "trtis" deleted
ls -al /ssd/gitlab/clara/devenv/sdk/deploy/clara-reference-workflow/test-data/00000000-0000-0000-0000-000000000000/ai-vnet/output
total 37644
drwxrwxr-x 2 root root 4096 Feb 25 00:35 .
drwxrwxr-x 4 gbae gbae 4096 Feb 25 00:35 ..
-rw-r--r-- 1 root root 324 Feb 25 00:35 recon.seg.mhd
-rw-r--r-- 1 root root 38535168 Feb 25 00:35 recon.seg.raw
The Clara reference pipeline has the following folder structure:
.
├── charts
│ ├── clara-workflow
│ │ ├── charts
│ │ ├── Chart.yaml
│ │ ├── _stage_values.yaml
│ │ ├── templates
│ │ │ ├── deployment.yaml
│ │ │ ├── _helpers.tpl
│ │ │ ├── NOTES.txt
│ │ │ ├── stages
│ │ │ │ ├── ai-vnet
│ │ │ │ │ ├── container.yaml
│ │ │ │ │ └── volume.yaml
│ │ │ │ └── template
│ │ │ │ ├── _container.yaml
│ │ │ │ └── _volume.yaml
│ │ │ ├── tests
│ │ │ │ └── test-connection.yaml
│ │ │ ├── volume-claim.yaml
│ │ │ └── volume.yaml
│ │ └── values.yaml
│ └── trtis
│ ├── charts
│ ├── Chart.yaml
│ ├── templates
│ │ ├── deployment.yaml
│ │ ├── _helpers.tpl
│ │ ├── NOTES.txt
│ │ ├── service.yaml
│ │ └── tests
│ │ └── test-connection.yaml
│ └── values.yaml
├── clara-wf
├── README.md
└── test-data
└── 00000000-0000-0000-0000-000000000000
└── ai-vnet
└── input
├── recon.mhd
└── recon.raw
Clara includes the clara-wf CLI tool to manage Helm charts for stages. Running clara-wf without arguments returns the following help information:
$ ./clara-wf
./clara-wf [command] [arguments...]
Command List
list_stages | list | ls : List available stages
create_stage | create | cs [stage name] : Create stage
delete_stage | delete | ds [stage name] : Delete stage
test_workflow | test [workflow id] [job id] : Test workflow
default args: ai-only 00000000-0000-0000-0000-000000000000
install_trtis : Deploy TRTIS service
uninstall_trtis : Delete TRTIS service
set_ngc_api_key | setkey : Reset NGC API key
Currently, only the ai-vnet stage is available now, with ‘ai-only’ pipeline as default.
Add the dicom-reader stage with the following commands (and expected output):
$ ./clara-wf ls
ai-vnet
$ ./clara-wf cs dicom-reader
Stage 'dicom-reader' is created at:
/ssd/gitlab/clara/devenv/sdk/deploy/clara-reference-workflow/charts/clara-workflow/templates/stages/dicom-reader
Default values for Stage 'dicom-reader' are added to:
/ssd/gitlab/clara/devenv/sdk/deploy/clara-reference-workflow/charts/clara-workflow/values.yaml
Test the pipelines with the following commands (and expected output):
$ ./clara-wf ls
ai-vnet
dicom-reader
$ ls charts/clara-workflow/templates/stages/dicom-reader
container.yaml volume.yaml
When ./clara-wf cs dicom-reader is executed, the charts/clara-workflow/templates/stages/template
directory is copied to charts/clara-workflow/templates/stages/dicom-reader, and the prefix _ is
removed from file names.
container.yaml
The file container.yaml in charts/clara-workflow/templates/stages/dicom-reader specifies the
dicom-reader clara container (stage).
Most settings can be used as they are, but modify this file to make container-specific settings, if needed. (See container v1 core specification) for more information.
{{- define "stages.dicom-reader.container" }}
{{- $stageName := "dicom-reader" }}
{{- $stageIndex := 0 }}
{{- $activeStages := (index .workflows .workflow.id) }}
{{- range $idx, $elem := $activeStages.stages }}
{{- if (eq $elem $stageName) }}{{- $stageIndex = $idx }}{{- end }}
{{- end }}
{{- $args := (index $activeStages.args $stageIndex) }}
{{- $waitName := (index $activeStages.waitLocks $stageIndex) }}
{{- $stage := (index .stages $stageName) -}}
image: "{{$stage.image.repository }}:{{$stage.image.tag }}"
imagePullPolicy: IfNotPresent
{{- if $stage.command }}
command: {{ $stage.command }}
{{- end }}
{{- if $args }}
args: {{ $args }}
{{- end }}
env:
- name: NVIDIA_CLARA_STAGENAME
value: "{{$stageName}}"
- name: NVIDIA_CLARA_RUNSIMPLE
value: "{{ .runSimple }}"
- name: NVIDIA_CLARA_TRTISURI
value: "{{ .trtisUri }}"
- name: NVIDIA_CLARA_APPDIR
value: "{{$stage.appDir }}"
- name: NVIDIA_CLARA_INPUTLOCK
value: "{{$stage.inputLock }}"
- name: NVIDIA_CLARA_INPUTS
value: "{{$stage.inputs }}"
- name: NVIDIA_CLARA_JOBID
value: "{{ .job.id }}"
- name: NVIDIA_CLARA_JOBNAME
value: "{{ .job.name }}"
- name: NVIDIA_CLARA_LOGNAME
value: "{{$stage.logName }}"
- name: NVIDIA_CLARA_LOCKDIR
value: "{{$stage.lockDir }}"
- name: NVIDIA_CLARA_LOCKNAME
value: "{{$stage.lockName }}"
- name: NVIDIA_CLARA_OUTPUTS
value: "{{$stage.outputs }}"
- name: NVIDIA_CLARA_TIMEOUT
value: "{{$stage.timeout }}"
- name: NVIDIA_CLARA_WAITNAME
value: "{{$waitName}}"
{{- end }}
volume.yaml
volume.yaml specifies volumes that are specific to the stage.
See Volumes and its APIs for more information.
{{- define "stages.dicom-reader.volume" }}
{{- $stageName := "dicom-reader" }}
{{- $stageIndex := 0 }}
{{- $activeStages := (index .workflows .workflow.id) }}
{{- range $idx, $elem := $activeStages.stages }}
{{- if (eq $elem $stageName) }}{{- $stageIndex = $idx }}{{- end }}
{{- end }}
{{- $activeStagesArr := (index .workflows .workflow.id).stages }}
{{- $activeStagesLen := (len $activeStagesArr) }}
{{- $firstStageName := (index $activeStagesArr 0 ) }}
{{- $lastStageName := (index $activeStagesArr (add $activeStagesLen -1)) }}
{{- $stage := (index .stages $stageName) -}}
# Input folder
- name: clara-payload-volume
mountPath: "{{$stage.appDir }}/{{$stage.mount.in.name }}"
{{- if (and .dicomAdapter.input (eq $firstStageName $stageName)) }}
subPath: "{{ .dicomAdapter.input }}"
{{- else }}
subPath: "{{ index$activeStages.ioFolders$stageIndex}}"
{{- end }}
# Output folder
- name: clara-payload-volume
mountPath: "{{$stage.appDir }}/{{$stage.mount.out.name }}"
{{- if (and .dicomAdapter.output (eq $lastStageName $stageName)) }}
subPath: "{{ .dicomAdapter.output }}"
{{- else }}
subPath: "{{ index$activeStages.ioFolders (add$stageIndex1) }}"
{{- end }}
# Locks folder (shared by containers)
- name: clara-payload-volume
mountPath: "{{$stage.appDir }}/locks"
subPath: "locks"
# Payload folder (shared by containers. For direct accessing to the payload of other containers.)
- name: clara-payload-volume
mountPath: "payloads"
{{- end }}
values.yaml
The files container.yaml and volume.yaml use Helm template to fill necessary values from values.yaml in the charts/clara-workflow/ folder.
Concrete values regarding pipeline and stage in values.yaml can be used in container.yaml and
volume.yaml by using $stages object (e.g., mountPath: "{{ $stage.appDir }}/{{ $stage.mount.out.name }}").
# Pipeline parameters
## Value used here should be one of key in 'workflows' object
workflow:
id: "ai-only" # Need to specify manually/programmatically
payloads:
hostPath: "" # Need to specify manually/programmatically
storage: 20Gi
# Job parameters
job:
id: "00000000-0000-0000-0000-000000000000" # Need to specify manually/programmatically
name: "Test Job" # Need to specify manually/programmatically
# DICOM Adapter parameters
## Set both values (input & output) to the empty string if you don't want to use DICOM Adapter
dicomAdapter:
input: "dicom-server" # First stage's input host path would be "/clara-io/clara-core/payloads/{{job.id}}/dicom-server"
output: "output" # Last stage's output host path would be "/clara-io/clara-core/payloads/{{job.id}}/output"
# Miscellaneous parameters
runSimple: "FALSE" # Need to specify manually/programmatically
trtisUri: "localhost:8000" # Need to specify manually/programmatically
# Pipeline definitions
## - Keys that work with DICOM Adapter should match with pipeline IDs used in DICOM Adapter
## - An item 'args' array should be an array of strings if the arguments exist (If not, specify the empty string)
workflows:
ct:
stages: ["dicom-reader", "recon", "ai-vnet", "dicom-writer"]
waitLocks: ["", "dicom-reader.lock", "recon.lock", "ai-vnet.lock"]
args: ["", "", "", ""]
ai:
stages: ["dicom-reader", "ai-livertumor", "dicom-writer"]
waitLocks: ["", "dicom-reader.lock", "ai-livertumor.lock"]
args: ["", "", ""]
dicom-reader-only:
stages: ["dicom-reader"]
waitLocks: [""]
args: [""]
ai-only:
stages: ["ai-vnet"]
waitLocks: [""]
args: [""]
recon-only:
stages: ["recon"]
waitLocks: [""]
args: [""]
dicom-writer-only:
stages: ["dicom-writer"]
waitLocks: [""]
args: [""]
# Stage definitions
stages:
...
##BEGIN_dicom-reader##
dicom-reader:
image:
repository: dicom-reader
tag: latest
mount:
in:
name: "input"
out:
name: "output"
stageName: "dicom-reader"
appDir: "/app"
inputLock: "/app/locks/input.lock"
inputs: "input/input.mhd"
logName: "/app/logs/dicom-reader.log"
outputs: "output/output.mhd"
lockDir: "/app/locks"
lockName: "dicom-reader.lock"
timeout: "300"
##END_dicom-reader##
...
The default stage values shown above come from charts/clara-workflow/_stage_values.yaml The
default workflow/stage values can be overridden when installed by Helm.
An excerpt from the clara-wf script (test_workflow method) shows how the values are overridden:
local workflow_id="ai-only"
local job_id="00000000-0000-0000-0000-000000000000"
[ -n "$1" ] && workflow_id="$1"
[ -n "$2" ] && job_id="$2"
# Build Clara container images locally (Need to update whenever new stages are added)
## ai-vnet
docker build -t ai-vnet -f ${TOP}/Applications/Operators/ai/app_vnet/Dockerfile ${TOP}/Applications/Operators/ai
# Deploy the Helm chart for the reference pipeline ('ai-only' pipeline is used below)
## - Do not set 'runSimple="TRUE"' when interacting with Clara Core
## - Set 'workflow.id' to other values (such as 'ct') to execute different workflow
## - Set 'workflow.payloads.hostPath' to host payloads path ({{workflow.payloads.hostPath}}/{{job.id}} would be payload folder for a job)
## - Set 'job.id' in UUID form (otherwise, execution will fail)
## - Set 'dicomAdapter.input' to the empty string if the first stage's input is not from DICOM Adapter output
## - Set 'dicomAdapter.output' to the empty string if you don't want to push the last stage's output to DICOM Adapter
## - Use '--dry-run --debug' option for debugging
## - Refer to https://helm.sh/docs/using_helm/#the-format-and-limitations-of-set for `--set` and `--set-string` options.
helm install --name test-wf --namespace "${CLARA_NAMESPACE}" \
--set-string runSimple="TRUE" \
--set-string workflow.id="${workflow_id}" \
--set-string workflow.payloads.hostPath="${SCRIPT_DIR}/test-data" \
--set-string job.id="${job_id}" \
--set-string job.name="Test Job" \
--set-string trtisUri="${NVIDIA_CLARA_TRTISURI}" \
--set-string dicomAdapter.input="" \
--set-string dicomAdapter.output="" \
--wait \
${SCRIPT_DIR}/charts/clara-workflow
As you see from the command above, variables such as workflow.payloads.hostPath need to be
specified manually or programmatically to indicate correct information that is availble in runtime.
Once correct values are specified in container.yaml and volume.yaml for dicom-reader, you must
modify the above command to build container image for dicom-reader and prepare test data.
gbae@gigony:/ssd/gitlab/clara/devenv/sdk/deploy/clara-reference-workflow/test-data$ tree
.
├── 00000000-0000-0000-0000-000000000000
│ └── ai-vnet
│ └── input
│ ├── recon.mhd
│ └── recon.raw
└── 11111111-1111-1111-1111-111111111111
└── dicom-reader
└── input
└── patient-id
└── study-uid
└── series-uid
└── SOP-uid.dcm
Finally, execute ./clara-wf test "dicom-reader-only" "11111111-1111-1111-1111-111111111111"
The procedures above add dicom-reader stage to the reference workflow. Once all stages needed for
CT or AI pipeline are added and each stage is tested with the clara-wf test command, Clara Core is
able to use the pipeline Helm chart and the helm install command can be executed programatically.