Creating an Application

Using Textual Representation and CLI Tools

Set the NGC API Key

Set the NGC API key to be able to sync microservices from NGC:

Get the NGC API Key - Refer to Generating Your NGC API Key.

$ ucf_app_builder_cli registry repo set-api-key -a <API-KEY>
Repository 'UCF' added

The command will print the repositories that are accessible using the API key.

List Available Microservices

To get the list of available microservices:

$ ucf_app_builder_cli service list
List of available services in local repository:
==============================================
ucf.svc.myservice

List of available services in UCF repository:
================================================
ucf.svc.botmaker.dialog-manager
ucf.svc.botmaker.speech-controller
ucf.svc.botmaker.speech-web-app
ucf.svc.botmaker.text-web-app
ucf.svc.riva.speech-skills

The command lists available microservices built locally as well as in NGC repository.

View Information of Microservices

To view information about microservices, run:

$ ucf_app_builder_cli service info -n ucf.svc.botmaker.speech-web-app
name: ucf.svc.botmaker.speech-web-app
specVersion: 2.0.0
chart: https://helm.ngc.nvidia.com/eevaigoeixww/ucf-11-ea-release/charts/botmaker-speech-web-app-2.0.0.tgz
description: BotMaker Speech Web App
type: msapplication
tags: []
keywords: []
publish: false
ciTrigger: false
egress:
-   name: speech-controller
    description: Speech Controller gRCPC endpoint
    protocol: TCP
    scheme: grpc
    mandatory: true
    data-flow: in-out
ingress:
-   name: webapp-ui
    description: Web App UI Endpoint
    scheme: http
    data-flow: in-out
    service: botmaker-speech-web-app-deployment-service
    port: 5001
    protocol: TCP
metadata: {}
version: 2.0.0
displayName: BotMaker Speech Web App
category:
    functional: Speech AI
    industry: General
secrets: []
buildToolVersion: 1.1.1

Parameters:
-------------
               debug: (boolean), Start bot in debug mode [Mandatory:False]
            feedback: (boolean), Enables feedback buttons in web mode [Mandatory:False]
     gunicornWorkers: (string ), Number of Gunicorn Workers for FM Server [Mandatory:False]
     gunicornTimeout: (string ), Gunicorn Timeout for FM Server [Mandatory:False]

Compliance Info:
---------------------
Report Generated on 2022-11-22 18:24:51 (UTC) using MSBuilder v1.1.1
Development compliance (Mandatory) 100.0%
Development compliance (Optional) 66.66%

Based on this information, users can choose the microservices to use and how to use them in the applications - setting parameters, connecting endpoints etc.,

Adding --show-docs to the above command, will print detailed documentation for the microservice.

 $ ucf_app_builder_cli service info -n ucf.svc.botmaker.speech-web-app --show-docs
...
Documentation:
--------------
╔═══════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╗
║                                              ucf.svc.botmaker.speech-web-app                                              ║
╚═══════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╝


                                                         Description

BotMaker Speech Web App

Sample web application that communicates with Speech Controller and can be used to interact with speech based bot using web
browser.


                                                            Usage

                                                           Params:

┌───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ botmaker-webapp:                                                                                                          │
│   debug: false                                                                                                            │
│   # Show debug info for bot in Web App                                                                                    │
│   feedback: true                                                                                                          │
│   # Enables feedback buttons in web mode                                                                                  │
│   gunicornWorkers: "1"                                                                                                    │
│   # Number of Gunicorn Workers for Web App                                                                                │
│   gunicornTimeout: "120"                                                                                                  │
│   # Gunicorn Timeout for Web App                                                                                          │
└───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘

                                                        Connections:

┌───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ connections:                                                                                                              │
│   botmaker-webapp/speech-controller: speech-controller/grpc-api                                                           │
└───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘


                                                         Performance

NA


                                                     Supported Platforms

x86 dGPU


                                                   Deployment requirements

 1 Make sure k8s foundational services are running


                                                           License

NVIDIA Proprietary License


                                                 Known Issues / Limitations

NA


                                                         References

NA

Create Application Boilerplate

Now that the microservices that will be required have been chosen, we can start creating the application.

We will first create a boilerplate and then start updating it to match the reference application Update the Reference Application.

Run the following command to generate a application boilerplate:

$ ucf_app_builder_cli app create my-application
UCF Application graph boilerplate generated at my-application.yaml (Doc: README.md)

This will generate a new file my-application.yaml along with README.md for the application documentation.

The file is a template with contents:

specVersion: 2.0.0

version: 0.0.1

doc: README.md

name: my-application

description: Description for my-application

dependencies:
- ucf.svc.microserviceA:0.0.1
- ucf.svc.microserviceB:0.0.1

components:
- name: microserviceA
  type: ucf.service.microserviceA
  parameters:
    paramA1: 100
    paramA2:
    - name: test
  secrets:
    some-secret-name: some-secret
- name: microserviceB
  type: ucf.service.microserviceB
  parameters:
    paramB1: false
    paramB2:
    - name: test

connections:
  microserviceA/http: microserviceB/http

vaultAgent:
  auth:
    path: auth/jwt/authMountPath
    type: jwt
    jwt:
      audience: https://vault-service:443
      role: auth-role
  role: auth-role
  namespace: vault-namespace
  service: https://vault-service/

# certificates:
#   devCerts:
#     file: certificates.yaml
#     addToHelmChart: false

secrets:
  some-secret:
    vaultAgent:
      path: secret/mount/path
      template:
        type: kv
        key: KEY_IN_SECRET

  # some-secret:
  #   k8sSecret:
  #     secretName: <k8sSecretName>
  #     key: <keyInK8sSecret>

  # some-secret:
  #   secretsStoreCsi:
  #     providerClassName: <providerClassName>
  #     objectName: <objectName>

  # some-secret:
  #   certificate:
  #     certName: <certificateName>
  #     type: <certificate/privateKey/issuingCA>

Note

App name shouldn’t contain spaces

Update App Information

Update the basic information for the application like name, short description etc. For more information on the fields refer to Basic Information.

After updating, the section would look like:

specVersion: 2.0.0

version: 1.13.1

doc: README.md

name: ucf-bot-app

description: UCF application for a bot

Update Dependencies

Add the microservices to be used in the application to the dependencies section. This information is required to fetch desired version of microservices when building the application.

A list of microservices and their desired versions must be added to this section. The format is a list of strings <microservice-type>:<microservice-version>. The microservice version can be obtained from the service info command described above.

To view all available versions of a microservice, run:

$ ucf_app_builder_cli service versions -n ucf.svc.botmaker.speech-web-app
List of versions for service in UCF repository:
===============================================
2.0.0

<microservice-version> can be a fixed version or in the format of NPM version range spec https://github.com/npm/node-semver#ranges

After updating the section would look like:

dependencies:
- ucf.svc.riva.speech-skills:2.0.0
- ucf.svc.botmaker.speech-web-app:>=2.0.0 && < 3.0.0
- ucf.svc.botmaker.speech-controller:2.0.x
- ucf.svc.botmaker.dialog-manager:2.0.0

In the above example, when the application is built, the tools will fetch the highest version that matches the version specification for that microservice.

  • For ucf.svc.riva.speech-skills and ucf.svc.botmaker.dialog-manager, version 2.0.0 will be picked

  • For ucf.svc.botmaker.speech-web-app, if the available versions are 1.0.0, 2.0.0, and 3.0.0, version 3.0.0 will be picked

  • For ucf.svc.botmaker.speech-controller, if the available versions are 1.0.0, 2.0.0 and 2.1.0, version 2.0.0 will be picked

This allows latest compatible versions of microservices to be picked when building applications without having to update the application graph file.

UCF Microservices follow semantic versioning - <MAJOR>.<MINOR>.<PATCH>

  • Changes in <MAJOR> part of version indicate major changes with possible breakage in backward compatibility.

  • Changes in <MINOR> part of version indicate enhancements done to the microservice with backward compatibility preserved.

  • Changes in <PATCH> part of version indicate minor changes like bug fixes done to the microservice with backward compatibility preserved.

Add and Configure Microservices

Next, we will add microservices to the application and configure them i.e set their parameters and set secrets. This is done by adding microservices to the components section.

For example, ucf.svc.botmaker.speech-web-app can be added to the application, using:

- name: botmaker-webapp
  type: ucf.svc.botmaker.speech-web-app
  parameters:
    debug: false
    gunicornTimeout: "60"
    imagePullSecrets:
    - name: ngc-docker-reg-secret

The parameters must be set according to the parameter schema for the microservice which can be obtained from the service info command output. For example, for the ucf.svc.botmaker.speech-web-app:

$ ucf_app_builder_cli service info -n ucf.svc.botmaker.speech-web-app
...
Parameters:
-------------
              speech: (boolean), Speech Web App or Text Web App [Mandatory:False]
               debug: (boolean), Start bot in debug mode [Mandatory:False]
            feedback: (boolean), Disables feedback buttons in web mode [Mandatory:False]
     gunicornWorkers: (string ), Number of Gunicorn Workers for FM Server [Mandatory:False]
     gunicornTimeout: (string ), Gunicorn Timeout for FM Server [Mandatory:False]
...

Here, debug is a boolean parameter and gunicornTimeout is a string parameter. Parameters that are marked mandatory must be specified. Some microservices have nested parameters as seen in the example below.

UCF application build tools will validate these parameters. During application build, the build tools will throw an error if mandatory parameters are not set or they do not match parameter schema. For example if debug is set to "123":

Validating application ...
debug:'123' is not of type 'boolean'
Warning!!! parameter validation failed for microservice instance botmaker-webapp
Application validation failed

For microservices whose helm charts are built using UCF MS Builder Tool, some parameters are implicitly added. These parameters are not shown in the service info output. For example, imagePullSecrets is an implicit parameter. Implicitly added parameters are described in the next section.

Implicitly Added Parameters

As mentioned in previous section, microservices whose helm charts are generated using the UCF MS Builder Tool have some parameters added implicitly.

Following is an example of how to set these parameters along with their details:

- name: <svc-name>
  type: <svc-ucf-type>
  parameters:
    imagePullSecrets:        # List of K8S secrets to be used for pulling images required by the MS (Since 1.0 EA)
    - name: ngc-docker-reg-secret

    ucfVisibleGpus: [1, 2]   # List of GPUs that should be made visible to the containers in the MS. Internally sets NVIDIA_VISIBLE_GPUS env var (Since 1.0 EA)

    resources:               # Resource requests and limits to set on the containers in the MS. (Since 1.0 EA)
      requests:              # Refer to - https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.22/#resourcerequirements-v1-core
        cpu: 1
      limits:
        nvidia.com/gpu: 1

    podAnnotations:          # Annotations to set on the pods belonging to the MS (Since 1.1 EA)
        annotation-name: annotation-value

    podLabels:               # Labels to set on the pods belonging to the MS (Since 1.1 EA)
        label-name: label-value

    extraPodVolumes:         # List of extra volumes to add to all the pods belonging to the MS (Since 1.1 EA)
    - name: extra-vol1       # Refer to https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.22/#volume-v1-core
      emptyDir: {}

    extraPodVolumeMounts:    # List of extra volume mounts to add to all the containers of all the pods belonging to the MS (Since 1.1 EA)
    - name: extra-vol1:
      mountPath: /tmp/extra-vol-dir

    svcAnnotations:          # Annotations to set on the services belonging to the MS (Since 1.1 EA)
        annotation-name: annotation-value

    svcLabels:               # Labels to set on the services belonging to the MS (Since 1.1 EA)
        label-name: label-value

    global:
        ucfGlobalEnv:        # List of environment variables to set in all containers of all the pods belonging to the MS (Since 1.1 EA)
        - name: SOME_ENV_VAR_NAME
          value: someEnvVarValue

Configuring Secrets

All mandatory secrets for each microservice must be specified. The secrets requirement can be viewed using the service info command.

Refer to Secrets Management for more information on how secrets work in UCF and how to configure them.

Connect Microservice Endpoints

Next we connect microservices together. Microservices that have mandatory Egress endpoints must be connected to another meaningful microservice that have corresponding and compatible Ingress endpoints that provide the necessary functionality required.

The endpoints of a microservice can be obtained from the service info command output.

Connections are specified as an object with top-level key connections in the application yaml. The connections are in the format:

connections:
  <egress-endpoint-component-name1>/<egress-endpoint-name1>: <ingress-endpoint-component-name1>/<ingress-endpoint-name1>
  <egress-endpoint-component-name2>/<egress-endpoint-name2>: <ingress-endpoint-component-name2>/<ingress-endpoint-name2>

In case the egress endpoint supports multiple connections (multi: true), multiple ingress endpoints can be connected to it using array style format:

connections:
  <egress-endpoint-component-name>/<egress-endpoint-name>: [
      <ingress-endpoint-component-name1>/<ingress-endpoint-name1>,
      <ingress-endpoint-component-name2>/<ingress-endpoint-name2>
  ]

UCF application build tools will validate the endpoints and connections. If a mandatory Egress endpoint is not connected or an incase of an invalid connection, the tools will throw an error.

Add Detailed Documentation

Add any detailed documentation required for building the application and then deploying the built helm chart to the README file accompanying the application graph file.

Value Spec File for Deployment

Application developers can choose a value spec file as an optional input while building an application, and the file specifies how values for deployment are overridden. The file is in multi-doc YAML format and each doc defines a value specification applied to certain microservice instances in the application.

Below is a simple example of value spec file

spec:
  namespace: my-application
---
spec:
  serviceAccount:
    name: my-sa
    create: True
usedBy: ["dialog-manager", "speech-controller"]

“usedBy” is optional for a value specification and all service instances in the application are applicable to it when “usedBy” is absent.

The following table lists all the values supported in a value spec.

Field

Description

namespace

Namespace to which the service is expected to be deployed

serviceAccount

serivce account information for a microservice

nodeSelector

node labels to select the nodes on which a Pod can be scheduled, more information in <https://kubernetes.io/docs/reference/kubernetes-api/workload-resources/pod-v1/#scheduling>

tolerations

rules for allowing a Pod to be scheduled on nodes with matching taints, more information in <https://kubernetes.io/docs/reference/kubernetes-api/workload-resources/pod-v1/#scheduling>

affinity

rules for constraining which nodes a Pod can be scheduled on, more information in <https://kubernetes.io/docs/reference/kubernetes-api/workload-resources/pod-v1/#scheduling>

storageClassName

storage class name used for the persistent volume claims

Using UCF Studio

Refer to Create an Reference Application for an example of creating an application using UCF Studio.

Building an Application

UCF Applications can be built using either the UCF Application Builder CLI tool or the UCF Studio GUI.

Using CLI Tools

Existing application graph files can be built using the following command:

$ ucf_app_builder_cli app build app.yaml
2022-12-05 14:44:52,464 - AppBuilder - INFO - Syncing any missing service versions to cache...
2022-12-05 14:44:52,507 - AppBuilder - INFO - Validating application ...
2022-12-05 14:44:52,517 - AppBuilder - INFO - Building application ucf-bot-app-1.13.1 ...
2022-12-05 14:45:00,122 - AppBuilder - INFO - Application Information written to ucf-bot-app-1.13.1/app_info.yaml
2022-12-05 14:45:00,122 - AppBuilder - INFO - Application compliance report generated at ucf-bot-app-1.13.1/compliance_report.json
2022-12-05 14:45:00,123 - AppBuilder - INFO - Application Helm Chart generated in ucf-bot-app-1.13.1
2022-12-05 14:45:00,123 - AppBuilder - INFO - App built

Parameter files can be applied using the following command:

$ ucf_app_builder_cli app build app.yaml params1.yaml params2.yaml

A value spec file can be specified using ‘-v’

$ ucf_app_builder_cli app build app.yaml params1.yaml params2.yaml -v value_spec.yaml

Application build performs several validations including any syntax errors, incorrect parameters, invalid connections and fails in case of any issues:

$ ucf_app_builder_cli app build app.yaml
2022-12-05 14:46:16,846 - AppBuilder - INFO - Syncing any missing service versions to cache...
2022-12-05 14:46:16,888 - AppBuilder - INFO - Validating application ...
2022-12-05 14:46:16,891 - AppBuilder - ERROR - Mandatory endpoint 'speech-controller' not set for microservice instance speech-web-app
2022-12-05 14:46:16,899 - AppBuilder - ERROR - Application validation failed

Using UCF Studio

Refer to Create an Reference Application for an example of building an application using UCF Studio.

Build Output

Building an application generates the following:

  • A deployable helm chart at the location mentioned by the command output e.g. ucf-bot-app-1.13.1

  • Application information file with some information useful for deploying the application e.g. ucf-bot-app-1.13.1/app_info.yaml. Currently this includes:

    • List of secrets used in the application and their details. This can be used to setup all the secrets prior to deploying the application.

    • List of metrics and their details exposed by microservices used in the application. This can be used to create any Prometheus rules or alerts or HPA configurations.

    • Sample for this file:

      metrics:
      - name: nv_inference_count
        description: No. of inference requests executed by the server
        labels: [ model, version ]
        microservice: ucf.svc.riva.speech-skills v2.0.0
      
      secrets:
      - name: my-ngc-api-key
        description: Secret for NGC API key
        type: Kubernetes Secret
        secretName: ngc-api-key-secret
        secretKey: NGC_API_KEY
        usedBy:
        - ucf.svc.riva.speech-skills v2.0.0
        - ucf.svc.botmaker.speech-controller v2.0.0
        - ucf.svc.botmaker.dialog-manager v2.0.0
      
  • A value override yaml file which carries a subset of values applicable to the application helm charts and can be used to override the values during installation

  • Combined compliance report of all the microservices added to the application e.g. ucf-bot-app-1.13.1/compliance_report.json. Sample for this file:

    {
        "name": "ucf-bot-app",
        "version": "1.13.1",
        "time-utc": "2023-03-13 13:21:33",
        "appbuilder-version": "2.0.0",
        "compliance-development-mandatory-overall": 100.0,
        "compliance-development-optional-overall": 49.99,
        "compliance-ms": [
            {
                "name": "ucf.svc.riva.speech-skills",
                "version": "2.0.0",
                "compliance-development-mandatory": 100.0,
                "compliance-development-optional": 66.66
            },
            {
                "name": "ucf.svc.botmaker.speech-web-app",
                "version": "2.0.0",
                "compliance-development-mandatory": 100.0,
                "compliance-development-optional": 66.66
            },
            {
                "name": "ucf.svc.botmaker.speech-controller",
                "version": "2.0.0",
                "compliance-development-mandatory": 100.0,
                "compliance-development-optional": 33.33
            },
            {
                "name": "ucf.svc.botmaker.dialog-manager",
                "version": "2.0.0",
                "compliance-development-mandatory": 100.0,
                "compliance-development-optional": 33.33
            }
        ]
    }
    

Deploy an Application

Building an UCF application generates a deployable helm chart. A simple way to deploy it is using the helm install command:

$ helm install <release-name> <app-output-helm-chart>

Some microservices expose their functionality to external users using the NodePort service type. Exposing NodePort services may not be possible or maybe tricky to expose in Kubernetes provided by some cloud service providers. In this case global.useLoadBalancerInsteadOfNodePort parameter maybe used to override the NodePort service type to LoadBalancer. This is only possible for microservices that are built using UCF 1.5 or later. An example of using the option is:

$ helm install <release-name> <app-output-helm-chart> –set=global.useLoadBalancerInsteadOfNodePort=true

Users may also have their own of deploying the application charts.

The application README which gets copied to the output helm chart may have additional pre-requisites/steps required to deploy the application.

Some applications depend on core services like Vault Agent Injector, Secrets Store CSI Driver. Users must deploy these services before deploying the applciation. The application information generated during app build might be helpful here.