Kubernetes Apps

Kubernetes add-ons were introduced in an older version of NVIDIA Base Command Manager (at that time still called Bright Cluster Manager) version 8.1. Add-ons could be managed in that version as part of the addons submode of the kubernetes mode in cmsh. The feature later (in version 8.2) was expanded into the Kubernetes Applications & Groups feature. Kubernetes Applications & Groups, less formally called app groups, can be accessed via the appgroups submode of cmsh:

Example

root@basecm11 ~# cmsh
[basecm11]% kubernetes
[basecm11->kubernetes[default]]% appgroups
[basecm11->kubernetes[default]->appgroups]% list
Name (key)   Applications
------------ ------------------------------
system       <13 in submode>
[basecm11->kubernetes[default]->appgroups]%

Versions from 9.2 onward started deploying operators as Helm charts, such as for the NVIDIA GPU operator, Run:ai, and NetQ. The remaining apps have since been migrated to equivalent Helm charts except for Flannel.

The legacy version 8.1 addons mode parameters are now accessed from version 8.2 onward via a default system app group instance. The system instance is accessed in the appgroups submode.

Example

[basecm11->kubernetes[default]->appgroups]% use system
[basecm11->kubernetes[default]->appgroups[system]]% show
Parameter                        Value
-------------------------------- ------------------------------------------------
Name                             system
Revision
Enabled                          yes
applications                     <13 in submode>
[basecm11->kubernetes[default]->appgroups[system]]% applications
[basecm11->kubernetes[default]->appgroups[system]->applications]% list
Name (key)          Format Enabled
------------------- ------ -------
bootstrap           Yaml   yes
calico              Yaml   yes
flannel             Yaml   no
kubestatemetrics    Yaml   yes
metrics_server      Yaml   yes
root                Yaml   yes

A Kubernetes application can span multiple namespaces. A name in appgroups therefore only exists to group logically-related applications. Each application contains a YAML configuration file, which BCM synchronizes to the Kubernetes API.

The default system app group is pre-defined. Other app groups can be created as needed. For example, an app group called monitoring could be created to group applications for running Prometheus, node exporters, and anything else related to exposing or viewing Prometheus metrics.

Toggling the Enable parameter of an app group enables or disables all of its application components in Kubernetes. Finer-grained control is possible within the applications mode level, by toggling the enabled parameter per application component instance. For example, within the calico application component instance:

Example

[basecm11->kubernetes[default]->appgroups[system]->applications]% use calico
[basecm11->kubernetes[default]->appgroups[system]->applications[calico]]% show
Parameter                        Value
-------------------------------- ------------------------------------------------
Name                             calico
Revision
Format                           Yaml
Enabled                          yes
Config                           <244KiB>
Environment                      <3 in submode>
Exclude list snippets            <2 in submode>

A large YAML configuration file for each application component instance can be configured via the Config parameter property, using the set option of cmsh. This opens up a text editor and allows the environment variables in the YAML configuration file to be managed.

Exclude list snippets are short exclude lists that can be set up for Kubernetes apps computing within the excludelistsnippets submode. They are used to prevent BCM software image updates from overwriting the provisioned files or directories of the container image that are important to the associated Kubernetes application.

Using exclude list snippets within an excludelistsnippets submode is discussed in detail in section 4.3.1 of the Cloudbursting Manual. Similar to the case of Kubernetes apps images, in cloud computing exclude list snippets are used to prevent overwriting of the provisioned files and directories of cloud images.

Environment entries can be set via the Environment submode. Environment entries are similar to environment variables, and are used to replace variables inside the YAML configuration file. Environment entries can be added to the environment as well, if the Nodes environment value inside the Environment submode is set to yes.

Example

[basecm11->kubernetes[default]->appgroups[system]->applications[calico]]% environment
[basecm11->kubernetes[default]->appgroups[system]->applications[calico]->environment]% list
Name (key)             Value            Nodes environment
---------------------- ---------------- ------------------
calico_typha_replicas  0                no
calico_typha_service   none             no
head_node_internal_ip  10.141.255.254   no