Creating an Application#
Using Textual Representation and CLI Tools#
Set the NGC API Key#
Set the NGC API key to be able to sync microservices from NGC: Make sure you have followed the prerequisites first for generating your NGC Personal Key.
$ ucf_app_builder_cli registry repo set-api-key -a <API-KEY>
Checking which NGC UCS teams you have access to
Adding NGC Catalog team ace
Repository 'catalog-ace' added
Adding NGC Catalog team ucs-ms
Repository names that begin with “catalog-” indicate that they exist in NGC Unified Catalog as opposed to NGC Private Registry. The NGC Unified Catalog consists of artifacts that are either fully public, or that require some product enablement (subscription) such as ace-ea or nvaie to get access to certain artifacts.
The command will print the repositories that are accessible using the API key.
List Available Microservices#
To get the list of available microservices:
$ ucf_app_builder_cli service list
List of available services in local repository:
==============================================
ucf.svc.myservice
List of available services in UCF repository:
================================================
ucf.svc.botmaker.dialog-manager
ucf.svc.botmaker.speech-controller
ucf.svc.botmaker.speech-web-app
ucf.svc.botmaker.text-web-app
ucf.svc.riva.speech-skills
The command lists available microservices built locally as well as in NGC repository.
View Information of Microservices#
To view information about microservices, run:
$ ucf_app_builder_cli service info -n ucf.svc.botmaker.speech-web-app name: ucf.svc.botmaker.speech-web-app specVersion: 2.5.0 chart: https://helm.ngc.nvidia.com/eevaigoeixww/ucf-11-ea-release/charts/botmaker-speech-web-app-2.0.0.tgz description: BotMaker Speech Web App type: msapplication tags: [] keywords: [] publish: false ciTrigger: false egress: - name: speech-controller description: Speech Controller gRCPC endpoint protocol: TCP scheme: grpc mandatory: true data-flow: in-out ingress: - name: webapp-ui description: Web App UI Endpoint scheme: http data-flow: in-out service: botmaker-speech-web-app-deployment-service port: 5001 protocol: TCP metadata: {} version: 2.0.0 displayName: BotMaker Speech Web App category: functional: Speech AI industry: General secrets: [] buildToolVersion: 1.1.1 Parameters: ------------- debug: (boolean), Start bot in debug mode [Mandatory:False] feedback: (boolean), Enables feedback buttons in web mode [Mandatory:False] gunicornWorkers: (string ), Number of Gunicorn Workers for FM Server [Mandatory:False] gunicornTimeout: (string ), Gunicorn Timeout for FM Server [Mandatory:False] Compliance Info: --------------------- Report Generated on 2022-11-22 18:24:51 (UTC) using MSBuilder v1.1.1 Development compliance (Mandatory) 100.0% Development compliance (Optional) 66.66%
Based on this information, users can choose the microservices to use and how to use them in the applications - setting parameters, connecting endpoints etc.,
Adding --show-docs to the above command, will print detailed documentation for the microservice.
$ ucf_app_builder_cli service info -n ucf.svc.botmaker.speech-web-app --show-docs
...
Documentation:
--------------
╔═══════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╗
║ ucf.svc.botmaker.speech-web-app ║
╚═══════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╝
Description
BotMaker Speech Web App
Sample web application that communicates with Speech Controller and can be used to interact with speech based bot using web
browser.
Usage
Params:
┌───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ botmaker-webapp: │
│ debug: false │
│ # Show debug info for bot in Web App │
│ feedback: true │
│ # Enables feedback buttons in web mode │
│ gunicornWorkers: "1" │
│ # Number of Gunicorn Workers for Web App │
│ gunicornTimeout: "120" │
│ # Gunicorn Timeout for Web App │
└───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
Connections:
┌───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ connections: │
│ botmaker-webapp/speech-controller: speech-controller/grpc-api │
└───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
Performance
NA
Supported Platforms
x86 dGPU
Deployment requirements
1 Make sure k8s foundational services are running
License
NVIDIA Proprietary License
Known Issues / Limitations
NA
References
NA
Create Application Boilerplate#
Now that the microservices that will be required have been chosen, we can start creating the application.
We will first create a boilerplate and then start updating it to match the reference application Update the Reference Application.
Run the following command to generate a application boilerplate:
$ ucf_app_builder_cli app create my-application
UCF Application graph boilerplate generated at my-application.yaml (Doc: README.md)
This will generate a new file my-application.yaml along with README.md for the application documentation.
The file is a template with contents:
specVersion: 2.5.0 version: 0.0.1 doc: README.md name: my-application description: Description for my-application dependencies: - ucf.svc.microserviceA:0.0.1 - ucf.svc.microserviceB:0.0.1 components: - name: microserviceA type: ucf.service.microserviceA parameters: paramA1: 100 paramA2: - name: test secrets: some-secret-name: some-secret - name: microserviceB type: ucf.service.microserviceB parameters: paramB1: false paramB2: - name: test connections: microserviceA/http: microserviceB/http vaultAgent: auth: path: auth/jwt/authMountPath type: jwt jwt: audience: https://vault-service:443 role: auth-role role: auth-role namespace: vault-namespace service: https://vault-service/ # certificates: # devCerts: # file: certificates.yaml # addToHelmChart: false secrets: some-secret: vaultAgent: path: secret/mount/path template: type: kv key: KEY_IN_SECRET # some-secret: # k8sSecret: # secretName: <k8sSecretName> # key: <keyInK8sSecret> # some-secret: # secretsStoreCsi: # providerClassName: <providerClassName> # objectName: <objectName> # some-secret: # certificate: # certName: <certificateName> # type: <certificate/privateKey/issuingCA>
Note
App name shouldn’t contain spaces
Update App Information#
Update the basic information for the application like name, short description etc. For more information on the fields refer to Basic Information.
After updating, the section would look like:
specVersion: 2.5.0 version: 1.13.1 doc: README.md name: ucf-bot-app description: UCF application for a bot
Update Dependencies#
Add the microservices to be used in the application to the dependencies section. This information is required to
fetch desired version of microservices when building the application.
A list of microservices and their desired versions must be added to this section. The format is a list of strings <microservice-type>:<microservice-version>.
The microservice version can be obtained from the service info command described above.
To view all available versions of a microservice, run:
$ ucf_app_builder_cli service versions -n ucf.svc.botmaker.speech-web-app
List of versions for service in UCF repository:
===============================================
2.0.0
<microservice-version> can be a fixed version or in the format of NPM version range spec npm/node-semver
After updating the section would look like:
dependencies:
- ucf.svc.riva.speech-skills:2.0.0
- ucf.svc.botmaker.speech-web-app:>=2.0.0 && < 3.0.0
- ucf.svc.botmaker.speech-controller:2.0.x
- ucf.svc.botmaker.dialog-manager:2.0.0
In the above example, when the application is built, the tools will fetch the highest version that matches the version specification for that microservice.
For
ucf.svc.riva.speech-skillsanducf.svc.botmaker.dialog-manager, version2.0.0will be pickedFor
ucf.svc.botmaker.speech-web-app, if the available versions are1.0.0,2.0.0, and3.0.0, version3.0.0will be pickedFor
ucf.svc.botmaker.speech-controller, if the available versions are1.0.0,2.0.0and2.1.0, version2.0.0will be picked
This allows latest compatible versions of microservices to be picked when building applications without having to update the application graph file.
UCS Microservices follow Semantic Versioning - <MAJOR>.<MINOR>.<PATCH>. Pre-release versions are also supported as discussed at semver.org (such as 1.0.0-alpha, 1.0.0-alpha.1, 1.0.0-0.3.7, etc). The following are some important concepts to keep in mind when using Semantic Versioning:
Changes in
<MAJOR>part of version indicate major changes with possible breakage in backward compatibility.Changes in
<MINOR>part of version indicate enhancements done to the microservice with backward compatibility preserved.Changes in
<PATCH>part of version indicate minor changes like bug fixes done to the microservice with backward compatibility preserved.
Add and Configure Microservices#
Next, we will add microservices to the application and configure them i.e set their parameters and set secrets. This is done
by adding microservices to the components section.
For example, ucf.svc.botmaker.speech-web-app can be added to the application, using:
- name: botmaker-webapp
type: ucf.svc.botmaker.speech-web-app
parameters:
debug: false
gunicornTimeout: "60"
imagePullSecrets:
- name: ngc-docker-reg-secret
The parameters must be set according to the parameter schema for the microservice which can be obtained from the service info command output.
For example, for the ucf.svc.botmaker.speech-web-app:
$ ucf_app_builder_cli service info -n ucf.svc.botmaker.speech-web-app ... Parameters: ------------- speech: (boolean), Speech Web App or Text Web App [Mandatory:False] debug: (boolean), Start bot in debug mode [Mandatory:False] feedback: (boolean), Disables feedback buttons in web mode [Mandatory:False] gunicornWorkers: (string ), Number of Gunicorn Workers for FM Server [Mandatory:False] gunicornTimeout: (string ), Gunicorn Timeout for FM Server [Mandatory:False] ...
Here, debug is a boolean parameter and gunicornTimeout is a string parameter. Parameters that are
marked mandatory must be specified. Some microservices have nested parameters as seen in the example below.
UCS Application build tools will validate these parameters. During application build, the build tools will throw an error if mandatory parameters are not set or they do not match parameter schema. For example if debug is set to "123":
Validating application ...
debug:'123' is not of type 'boolean'
Warning!!! parameter validation failed for microservice instance botmaker-webapp
Application validation failed
For microservices whose helm charts are built using UCS MS Builder Tool, some parameters are implicitly added.
These parameters are not shown in the service info output. For example, imagePullSecrets is an implicit parameter.
Implicitly added parameters are described in the next section.
Implicitly Added Parameters#
As mentioned in previous section, microservices whose helm charts are generated using the UCS MS Builder Tool have some parameters added implicitly.
Following is an example of how to set these parameters along with their details:
- name: <svc-name>
type: <svc-ucf-type>
parameters:
imagePullSecrets: # List of K8S secrets to be used for pulling images required by the MS (Since 1.0 EA)
- name: ngc-docker-reg-secret
ucfVisibleGpus: [1, 2] # List of GPUs that should be made visible to the containers in the MS. Internally sets NVIDIA_VISIBLE_GPUS env var (Since 1.0 EA)
resources: # Resource requests and limits to set on the containers in the MS. (Since 1.0 EA)
requests: # Refer to - https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.22/#resourcerequirements-v1-core
cpu: 1
limits:
nvidia.com/gpu: 1
podAnnotations: # Annotations to set on the pods belonging to the MS (Since 1.1 EA)
annotation-name: annotation-value
podLabels: # Labels to set on the pods belonging to the MS (Since 1.1 EA)
label-name: label-value
extraPodVolumes: # List of extra volumes to add to all the pods belonging to the MS (Since 1.1 EA)
- name: extra-vol1 # Refer to https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.22/#volume-v1-core
emptyDir: {}
extraPodVolumeMounts: # List of extra volume mounts to add to all the containers of all the pods belonging to the MS (Since 1.1 EA)
- name: extra-vol1:
mountPath: /tmp/extra-vol-dir
svcAnnotations: # Annotations to set on the services belonging to the MS (Since 1.1 EA)
annotation-name: annotation-value
svcLabels: # Labels to set on the services belonging to the MS (Since 1.1 EA)
label-name: label-value
global:
ucfGlobalEnv: # List of environment variables to set in all containers of all the pods belonging to the MS (Since 1.1 EA)
- name: SOME_ENV_VAR_NAME
value: someEnvVarValue
Configuring Secrets#
All mandatory secrets for each microservice must be specified. The secrets requirement can be viewed using the service info command.
Refer to Secrets Management for more information on how secrets work in UCS Tools and how to configure them.
Connect Microservice Endpoints#
Next we connect microservices together. Microservices that have mandatory Egress endpoints must be connected to another meaningful microservice that have corresponding and compatible Ingress endpoints that provide the necessary functionality required.
The endpoints of a microservice can be obtained from the service info command output.
Connections are specified as an object with top-level key connections in the application yaml. The connections are in the format:
connections:
<egress-endpoint-component-name1>/<egress-endpoint-name1>: <ingress-endpoint-component-name1>/<ingress-endpoint-name1>
<egress-endpoint-component-name2>/<egress-endpoint-name2>: <ingress-endpoint-component-name2>/<ingress-endpoint-name2>
In case the egress endpoint supports multiple connections (multi: true), multiple ingress endpoints can be connected to it using array style format:
connections: <egress-endpoint-component-name>/<egress-endpoint-name>: [ <ingress-endpoint-component-name1>/<ingress-endpoint-name1>, <ingress-endpoint-component-name2>/<ingress-endpoint-name2> ]
UCS Application build tools will validate the endpoints and connections. If a mandatory Egress endpoint is not connected or an incase of an invalid connection, the tools will throw an error.
Add Detailed Documentation#
Add any detailed documentation required for building the application and then deploying the built helm chart to the README file accompanying the application graph file.
Value Spec File for Deployment#
Application developers can choose a value spec file as an optional input while building an application, and the file specifies how values for deployment are overridden. The file is in multi-doc YAML format and each doc defines a value specification applied to certain microservice instances in the application.
Below is a simple example of value spec file
spec:
namespace: my-application
---
spec:
serviceAccount:
name: my-sa
create: True
usedBy: ["dialog-manager", "speech-controller"]
“usedBy” is optional for a value specification and all service instances in the application are applicable to it when “usedBy” is absent.
The following table lists all the values supported in a value spec.
Field |
Description |
|---|---|
namespace |
Namespace to which the service is expected to be deployed |
serviceAccount |
serivce account information for a microservice |
nodeSelector |
node labels to select the nodes on which a Pod can be scheduled, more information in <https://kubernetes.io/docs/reference/kubernetes-api/workload-resources/pod-v1/#scheduling> |
tolerations |
rules for allowing a Pod to be scheduled on nodes with matching taints, more information in <https://kubernetes.io/docs/reference/kubernetes-api/workload-resources/pod-v1/#scheduling> |
affinity |
rules for constraining which nodes a Pod can be scheduled on, more information in <https://kubernetes.io/docs/reference/kubernetes-api/workload-resources/pod-v1/#scheduling> |
storageClassName |
storage class name used for the persistent volume claims |
Using UCS Studio#
Refer to Create an Reference Application for an example of creating an application using UCS Studio.
Building an Application#
UCS Applications can be built using either the UCS Application Builder CLI tool or the UCS Studio GUI.
Using CLI Tools#
Existing application graph files can be built using the following command:
$ ucf_app_builder_cli app build app.yaml
2022-12-05 14:44:52,464 - AppBuilder - INFO - Syncing any missing service versions to cache...
2022-12-05 14:44:52,507 - AppBuilder - INFO - Validating application ...
2022-12-05 14:44:52,517 - AppBuilder - INFO - Building application ucf-bot-app-1.13.1 ...
2022-12-05 14:45:00,122 - AppBuilder - INFO - Application Information written to ucf-bot-app-1.13.1/app_info.yaml
2022-12-05 14:45:00,122 - AppBuilder - INFO - Application compliance report generated at ucf-bot-app-1.13.1/compliance_report.json
2022-12-05 14:45:00,123 - AppBuilder - INFO - Application Helm Chart generated in ucf-bot-app-1.13.1
2022-12-05 14:45:00,123 - AppBuilder - INFO - App built
Application build performs several validations including any syntax errors, incorrect parameters, invalid connections and fails in case of any issues:
$ ucf_app_builder_cli app build app.yaml
2022-12-05 14:46:16,846 - AppBuilder - INFO - Syncing any missing service versions to cache...
2022-12-05 14:46:16,888 - AppBuilder - INFO - Validating application ...
2022-12-05 14:46:16,891 - AppBuilder - ERROR - Mandatory endpoint 'speech-controller' not set for microservice instance speech-web-app
2022-12-05 14:46:16,899 - AppBuilder - ERROR - Application validation failed
Parameter files#
Rather than specify the parameters in the application yaml, some or all params for a given app component may be provided via parameters (params) files. For example, given the app example in the section named Create Application Boilerplate, instead of supplying the params inside of the app YAML, we could put in the a params.yaml file like follows:
microserviceA:
paramA1: 100
paramA2:
- name: test
microserviceB:
paramB1: false
paramB2:
- name: test
Parameter files can be applied using the following command:
$ ucf_app_builder_cli app build app.yaml params1.yaml params2.yaml
You can also mix and match so that the app YAML contains some params defined but others provided in a params file. If a parameter is defined in both places, the one in the params file will take precedence. Furhtermore, multiple params files can be specified to provide a cascading means for overriding params. The parameter defined in the right most location will take precedence. For example, if you build an app with the following command-line:
ucf_app_builder_cli app build app.yaml params1.yaml params2.yaml
and some parameter P for some app compoment C is specified in the app.yaml and both params1.yaml and params2.yaml, the value for P from params2.yaml will be used.
As of version 2.8.1, the built-in “files” parameter in an app component’s params block is also supported in a params file.
Value spec file#
A value spec file can be specified using ‘-v’
$ ucf_app_builder_cli app build app.yaml params1.yaml params2.yaml -v value_spec.yaml
Using UCS Studio#
Refer to Create an Reference Application for an example of building an application using UCS Studio.
Build Output#
Building an application generates the following:
A deployable helm chart at the location mentioned by the command output e.g.
ucf-bot-app-1.13.1
Application information file with some information useful for deploying the application e.g.
ucf-bot-app-1.13.1/app_info.yaml. Currently this includes:List of secrets used in the application and their details. This can be used to setup all the secrets prior to deploying the application.
List of metrics and their details exposed by microservices used in the application. This can be used to create any Prometheus rules or alerts or HPA configurations.
Sample for this file:
metrics: - name: nv_inference_count description: No. of inference requests executed by the server labels: [ model, version ] microservice: ucf.svc.riva.speech-skills v2.0.0 secrets: - name: my-ngc-api-key description: Secret for NGC API key type: Kubernetes Secret secretName: ngc-api-key-secret secretKey: NGC_API_KEY usedBy: - ucf.svc.riva.speech-skills v2.0.0 - ucf.svc.botmaker.speech-controller v2.0.0 - ucf.svc.botmaker.dialog-manager v2.0.0
A value override yaml file which carries a subset of values applicable to the application helm charts and can be used to override the values during installation
Combined compliance report of all the microservices added to the application e.g.
ucf-bot-app-1.13.1/compliance_report.json. Sample for this file:{ "name": "ucf-bot-app", "version": "1.13.1", "time-utc": "2023-03-13 13:21:33", "appbuilder-version": "2.0.0", "compliance-development-mandatory-overall": 100.0, "compliance-development-optional-overall": 49.99, "compliance-ms": [ { "name": "ucf.svc.riva.speech-skills", "version": "2.0.0", "compliance-development-mandatory": 100.0, "compliance-development-optional": 66.66 }, { "name": "ucf.svc.botmaker.speech-web-app", "version": "2.0.0", "compliance-development-mandatory": 100.0, "compliance-development-optional": 66.66 }, { "name": "ucf.svc.botmaker.speech-controller", "version": "2.0.0", "compliance-development-mandatory": 100.0, "compliance-development-optional": 33.33 }, { "name": "ucf.svc.botmaker.dialog-manager", "version": "2.0.0", "compliance-development-mandatory": 100.0, "compliance-development-optional": 33.33 } ] }
Deploy an Application with Helm#
Building an UCS Application generates a deployable helm chart. A simple way to deploy it is using the helm install command:
$ helm install <release-name> <app-output-helm-chart>
Some microservices expose their functionality to external users using the NodePort service type. Exposing NodePort services
may not be possible or maybe tricky to expose in Kubernetes provided by some cloud service providers. In this case global.useLoadBalancerInsteadOfNodePort
parameter maybe used to override the NodePort service type to LoadBalancer. This is only possible for microservices that are built using
UCS 1.5 or later. An example of using the option is:
$ helm install <release-name> <app-output-helm-chart> –set=global.useLoadBalancerInsteadOfNodePort=true
Users may also have their own of deploying the application charts.
The application README which gets copied to the output helm chart may have additional pre-requisites/steps required to deploy the application.
Some applications depend on core services like Vault Agent Injector, Secrets Store CSI Driver. Users must deploy these services before deploying the applciation. The application information generated during app build might be helpful here.
Deploy an Application to NVCF#
UCS Tools allows you to build an app and deploy directly on NVCF as a cloud function. The following example demonstrates deploying an existing UCS Microservice ucf.svc.nvcf.image-segmentation:1.3.0 to NVCF.
Create the App Manifest#
Save the following to a file named app.yaml:
specVersion: '2.5.0' version: 0.0.2 doc: README.md name: image-segmentation-app description: Performs segmentation of a given image and is based on Deepstream 6.2 triton docker dependencies: - ucf.svc.nvcf.image-segmentation:1.3.0 components: - name: image-segmentation type: ucf.svc.nvcf.image-segmentation parameters: imagePullSecrets: - name: ngc-docker-reg-secret
Create the NVCF Configuration file for UCS Tools#
Save the following to a file named nvcf.yaml and set “Password” field within “DockerRegSecrets” to your NGC API KEY:
FunctionName: "image-segmentation-service" # NVCF function name InferenceUrl: "/grpc" # NVCF function interface URL. Use '/grpc' for gRPC functions InferencePort: "50000" # NVCF function port HelmChartServiceName: "image-seg-nvcf-segmentation-deployment-segmentation-service" # K8s service name exposed as function GpuType: "A10G" # Type of GPU Backend: "GFN" # Backend type MaxInstance: "1" # Maximum number of instances MinInstance: "1" # Minimum number of instances InstanceType: "ga10g_1.br20_2xlarge" # Type of instance DockerRegSecrets: - Name: "ngc-docker-reg-secret" Username: "NGC_CLI_API_KEY" Password: "<KEY>" #AdditionalSecrets: # - Name: "" # Key: "" # Value: "" #StorageProvisioner: # - Name: "" # Namespace: ""
Deploy the UCS App to NVCF#
First, build the app:
ucf_app_builder_cli app build app.yaml
Then run the nvcf-deploy subcommand. This will upload your app Helm chart to nv-ucf/staging, unless it already exists with the version you are building. Then, it will create the NVCF function and deploy it:
> ucf_app_builder_cli app nvcf-deploy app.yaml nvcf.yaml {'version': False, 'version_all': False, 'parser_name': 'app', 'help': 'Create a microservice from the application', 'app': 'app.yaml', 'nvcf': 'nvcf.yaml', 'subparser_name': 'nvcf-deploy'} 2024-03-27 14:52:01,668 - AppBuilder - INFO - Syncing any missing service versions to cache... 2024-03-27 14:52:01,674 - AppBuilder - INFO - Deploying application image-segmentation-app-0.0.2 on NVCF 2024-03-27 14:52:01,675 - AppBuilder - INFO - nvcf metadata: {'FunctionName': 'image-segmentation-service', 'InferenceUrl': '/grpc', 'InferencePort': '50000', 'HelmChartServiceName': 'image-seg-nvcf-segmentation-deployment-segmentation-service', 'GpuType': 'A10G', 'Backend': 'GFN', 'MaxInstance': '1', 'MinInstance': '1', 'InstanceType': 'ga10g_1.br20_2xlarge', 'DockerRegSecrets': [ {'Name': 'ngc-docker-reg-secret', 'Username': 'NGC_CLI_API_KEY', 'Password': '<KEY>'} ]} 2024-03-27 14:52:04,687 - MsBuilder - INFO - App started deploying on NVCF
Also check out the UCS A2F demo.