4. Clara Administration

4.1. Configuration

4.1.1. Clara Deploy SDK Platform

Configuration of the Clara Deploy SDK is done via the values.yaml file in the Clara Deploy SDK Helm chart $HOME/.clara/charts/clara/values.yaml.

Some of the values defined in values.yaml are constants used throughout the Helm chart and are not user-configurable.

The following are user-configurable values:

platformapiserver:
  pipelineTimeoutDefault: 600 # Default timeout in seconds for pipeline-jobs (when not supplied via the pipeline definition).
  pipelineTimeoutMaximum: 1800 # Maximum timeout in seconds for any pipeline-job to have.
  pipelineTimeoutGrace: 15 # Number of seconds given to a timed out pod once termination has started.
  maxStorageUsagePercentage: 80 # maximum storage to use on the Payload Volume, Setting near or above 85 may cause system instability.

  # Configuration for the Nvidia TensorRT Inference Server
  inferenceServerConfig:
    maxInstances: 1
    modelLimits: 8

  # Configuration for the Job Logs Archive
  jobLogsArchive:
    hostVolumePath: "/clara/job-logs-archive/"
    capacity: 10Gi

  # Configuration for Model Analyzer's Metrics storage
  metricsService:
    hostVolumePath: "/clara/repository/metrics"
    capacity: 250Mi

  # Configuration for the Model Storage Service
  modelService:
    hostVolumePath: "/clara/repository/models"
    capacity: 10Gi

  # Configuration for the payload service in the Clara Deploy SDK server.
  payloadService:

    # The path on the node running Clara Deploy SDK where payloads will be stored.
    # Every payload that is created by Clara Deploy SDK will exist as a directory inside
    # this path (e.g. "/clara/payloads/c780b2cbd26b41518e7ab4fbfb876f69").
    hostVolumePath: "/clara/payloads"

    # The disk capacity to reserve for the payload volume.
    capacity: 10Gi

  # Configuration for the pod cleaner service.
  podCleaner:

    # Time between cleaner runs
    podCleanerFrequencyInMinutes: 30

    # Time since creation before pods can be deleted, if succeeded or faulted
    # Above 60 minutes, because pipeline jobs are allowed to run for one hour
    podCleanerBufferInMinutes: 1

    # Pod Cleaner can be disabled by setting this flag to false.
    podCleanerEnable: true

  # Configuration for the payload cleaner daemon.
  # This will clean up Persistent Volumes, Claims as well as Residual Data.
  payloadCleaner:
    # Time between cleaner runs
    payloadCleanerFrequencyInMinutes: 10

    # Time since creation before payloads can be deleted, if succeeded or faulted
    # Above 60 minutes, because pipeline jobs are allowed to run for one hour
    payloadCleanerBufferInMinutes: 60

    # payload Cleaner can be disabled by setting this flag to false.
    payloadCleanerEnable: true

  # Configuration for the common volume that is mounted (read-only) to all pipeline
  # service containers.
  #
  # The path that is used to mount this volume in each of the deployed pipeline
  # services is provided to those service containers via the
  # NVIDIA_CLARA_SERVICE_DATA_PATH environment variable. For example, a TensorRT
  # Inference Service may be defined in the pipeline using the following command:
  #
  #   --models=$NVIDIA_CLARA_SERVICE_DATA_PATH/models
  #
  # to specify that the models repository used by the service will be stored
  # inside the 'models' subdirectory of this common volume. In the case of the
  # default hostVolumePath of "/clara/common", the path provided for the models
  # repository of the TRTIS container would then correspond to the host path of
  # "/clara/common/models".
  #
  # Since this volume is mounted read-only by services, it is expected that the
  # path to this volume already exists on the host and is populated with any
  # required data for the services that use it (such as models used by a TRTIS
  # service). If it does not exist at deployment time, it will be created and
  # will be empty.
  commonServiceVolume:

    # The path on the node running Clara Deploy SDK where the common volume will
    # be stored.
    hostVolumePath: "/clara/common"

    # The disk capacity to reserve for the common volume.
    capacity: 10Gi

  # Configuration for the service volume used by the Clara Deploy SDK server.
  #
  # Each service that is deployed by Clara Deploy SDK using a 'volume' connection
  # is provided with a persistent volume to which each of the volume connections
  # are mounted. For example, this volume connection:
  #
  #   volume:
  #   - name: VOlUME_PATH
  #     path: /var/www
  #
  # Will provide a volume mount inside the service container that corresponds to
  # the host path of (using the default hostVolumePath)
  #
  #   /clara/service-volumes/{serviceId}/var/www
  #
  # where {serviceId} is the unique ID generated by Clara Deploy SDK that was used
  # to deploy the service.
  serviceVolume:

    # The path on the node running Clara Deploy SDK where service volumes will be stored.
    # Every service that is deployed by Clara Deploy SDK will have a volume mounted in this
    # path (e.g. "/clara/service-volumes/c780b2cbd26b41518e7ab4fbfb876f69").
    hostVolumePath: "/clara/service-volumes"

    # The disk capacity to reserve for the service volume.
    capacity: 10Gi

####################################################
#      Configuration Values for Log Collection     #
####################################################
clara-log-collector:
  enabled: true
  # Use this path to set where aggregated logs will be stored
  # default: "/clara/log-archive"
  # claraLogPath:
  elasticSearch:
    enabled: false
    # When Elastic Search is enabled for log publishing these configurations must be set.
    # If Elastic Search is secured, provide Username and create secret using
    # kubectl create secret to store password and provide the secret name to this configuration.
    #host: localhost
    #port: 9200
    #scheme: http
    #user: es-user
    #passwordSecret: es-password-secret

4.1.2. Render Server

Configuration of the Render Server is done via the values.yaml file in the Render Server Helm chart $HOME/.clara/charts/clara-renderer/values.yaml.

Some of the values defined in values.yaml are constants used throughout the Helm chart and are not user-configurable.

The following are user-configurable values:

# Configuration for the render service in the Clara Deploy SDK server.
rsds:
  # The disk capacity reserved for the  datasets.
  storage: 10Gi
  # The path on the node running Clara Deploy SDK where the datasets will be stored.
  # The subfolder path will have the name jobname-stagename of the triggered job.
  hostDatasetPath: "/clara-io/datasets"

4.1.3. Management Console

Configuration of the Configuration of the Clara Managment Console is done via the values.yaml file in the Clara Console Helm chart $HOME/.clara/charts/clara-ux/values.yaml.

Some of the values defined in values.yaml are constants used throughout the Helm chart and are not user-configurable.

The following are user-configurable values:

# Configuration for storage (hostpath) for mongodb
mongodb:
  persistence:
    hostPath: /clara/ux

4.2. Start/Stop/Restart the platform

To remove the the Clara Deploy SDK deployment:

clara dicom stop
clara render stop
clara platform stop
clara console stop

To redeploy Clara:

clara platform start
clara dicom start
clara render start
clara console start

To restart Clara:

clara platform restart
clara dicom restart
clara render restart
clara console restart

Note

Restarting the platform would also require a restart of DICOM Adapter and Render Server.

4.2.1. Restarting the Operating System

When restarted, a default operating-system deployment may enable swap automatically, causing commands like “helm ls” and “kubectl get pods” to fail. In this case, you will need to disable swap before underlying services can be accessed:

sudo -i
swapoff -a
exit

After this command is executed, Clara services should start, but may take a few minutes.

4.3. Network Configuration

The containers expose data for the user to between the services. The main components with exposed ports are:

Component Port
DICOM Adapter 104
Dashboard UI 8000
Render Server 2050, 8080
Management Console 32002, 32003
Clara API 30031

These ports are not configurable, so be sure that nothing else is using these ports.

4.4. AI models for TensorRT Inference Server

For AI inference application, trained models need to be made available for the NVIDIA TensorRT Inference Server. For details on what models are supported, how to configure a model, and TRTIS model repository, please refer to official NVIDIA documentation at the following link: https://docs.nvidia.com/deeplearning/sdk/tensorrt-inference-server-guide/docs/.

Each model’s folder must be copied to the Clara Deploy host at the path, /clara/common/models.

4.5. Key Commands

To get the list of the deployments:

helm ls

To get the name of the pods running on the system:

kubectl get pods

Hint: The Clara pods will be the one with clara in the name.

To view the details of a pod:

kubectl get po [POD_NAME]
kubectl describe po [POD_NAME]

where the POD_NAME is acquired from the kubectl get pods command.

To view the container logs of the containers in the pod:

kubectl logs [POD_NAME] [CONTAINER_NAME]

where the POD_NAME is acquired from the kubectl get pods command and the CONTAINER_NAME can be retrieved from the kubectl describe pods [POD_NAME] command. If the pod contains one single container, the CONTAINER_NAME is sufficient.

To log into the containers:

kubectl exec -it [POD_NAME] --container [CONTAINER_NAME] -- /bin/sh

where the POD_NAME is acquired from the kubectl get pods command and CONTAINER_NAME can be retrieved from the kubectl describe pods [POD_NAME] command.