Autonomous Job Recovery (AJR) (air-gapped)#

Install AJR in air-gapped mode using the BCM cm-mission-control-setup wizard. The bundle must be present in the current directory and the local registry must be configured to pull images from the bundle.

  1. Download all AJR files and related packages with nmc-airgap tool. When downloading files, select different output directories for amd64 and arm64 architectures, e.g. bundle_amd64 and bundle_arm64.

  2. Copy all files from the bundle*/files/ subdirectories into a single directory:

    mkdir -p bundle/files
cp -u bundle_*/files/*deb bundle/files/

  3. Install the AJR dependency and plugin packages from the bundle on all BCM head nodes. Installation wizard will fail if dependency and plugin packages are not installed on every head node:

    dpkg -i bundle/files/cm-base-view-ajr-bcm11*.deb
dpkg -i bundle/files/cm-setup-ajr-*.deb

  4. Follow the AJR installation guide at Installation Guide for Autonomous Job Recovery with the following notes and modifications:

    Note

    In NMC version 2.3, step 2 still requires internet access to install the observability components. Full air-gapped installation will be available in the next release.

    In NVIDIA Mission Control version 2.3.1, observability components are downloaded by the nmc-airgap tool and are located in the files/ subdirectory of the bundle. These files were copied into bundle/files subdirectory at step 2. This location must be specified during the AJR air-gapped installation wizard execution as a source for the Alloy packages.

    4.1 Exporting desired version of product and Helm/container registry repositories

    Several variables must be defined before AJR patch is applied. These variables are:

    REGISTRY — your container registry URL. For a BCM-installed private registry use the master.cm.cluster:5000/nvidia/nv-mission-control address.

    AJR_FQDN — the fully qualified domain name matching your TLS certificate.

    HELM_REPO_URL — your Helm repository URL. In air-gapped mode of installation, AJR is installed directly from the Helm chart archive. At the step 4 of the “Before the Autonomous Job Recovery installation” section, use the file:// protocol with the full path to the directory containing warthog-1.5.0-patchN.tgz Helm chart package as a source for the HELM_REPO_URL variable, for example:

    # ls -la /root/ajr/bundle_amd64/helm
drwxr-xr-x 2 root root     38 Jun  8 11:34 .
drwxr-xr-x 5 root root     75 Jun  8 11:35 ..
-rw------- 1 root root 318527 Jun  8 11:34 warthog-1.5.0-patch5.tgz

# export HELM_REPO_URL="file:///root/ajr/bundle_amd64/helm"

    The Grafana Alloy Helm chart and container images are not configured through variables. In air-gapped mode the patch always installs the Alloy chart from the BCM internal OCI registry at oci://master.cm.cluster:5000/helm-charts/alloy and pulls the Alloy container images from master.cm.cluster:5000, where the nmc-airgap tool uploads them. No additional variables are required for Alloy.

    Note

    Important! file:// protocol specifies full path from the root of the filesystem and contains three forward slash characters

    Note

    Important! Do not apply variable overrides mentioned in the section 4 of the connected installation guide for AJR.

    4.2 Apply the air-gapped AJR patch. This patches the same files as the connected patch and additionally redirects the Grafana Alloy Helm chart and container images to the BCM internal OCI registry. Run it after exporting the variables above and before starting the installation wizard:

    echo "H4sICKvyOGoAA3BhdGNoLWFpcmdhcC5kaWZmAM1YbXPbuBH+zl+BSvG5rUTSchInVpObKI7S+MaxXDm59uYmwwFBiMIJJBgAtKNp+9+7AEiKkujruW2c6oNIAIvFs+8L9n8XxiwPY6yWHsEafY9CnRVhTDKflEqLLCiwJkv04gWazt56vu+jkGQhFwTzEBeFgpGvqC6LkLM4LNZ6KfLHweg4VExTv8BkhVNqqBxRwcuU5SrEpRa5yESpol9EHElKxA2V61DTrOBYww6AECV0gUuuVbDGGfcGg8E3OzzI6a336hXyH4+Go2do4B6vXnkIay1ZXGomcsAibxihYw8hRHMcc5qMkZYlNRMsAzBjz4dXJGkhAKOQ6zHKb4gMmAjzG5YwDA8/Y0oZdkTkWgoeLjDRNW9vsLv/YD798/n1h/lPB9uE9ZFXJefXlEiqlQWGkI9ynNExcElhOvE8dCvkasHFbX0kp9LSZmWe0qiNHNxh+Vsw250OrNvSgunWYOV2EbGc6f/ogA7IdxzXRbljj4c7NFcac3Ay4BrpdQFWAF+z/nJDcx3FZW2kEuzozNSTOI6Zzj73rAc+fzIEB7T/xv86PK3ygIr/GYdApvL8ynOOwxnBaoyeGsmdDn6D4BAnfoZzoLYCV/ta8rYJ4BwiMhgBpp8PWyuHn9rwgPeCpZW4XKQRBx3wMehoIayko8fPjKjuYWMNohbztZ9QTYmNOKYEKFPI7pC7SxH3kfzuM7sV8Sv0Ri8Qb2BnhrlqBF+x6hUhSGoRLlikxYrmkbJRGzkvWFKWJeA6vtngWwKnpNOT4egJaMk9nUtUGoVsl144pZ5fvp3VK0pLnKbgkdNaYR/mH6f1alzGMPmDiNX2su+WiSCrgum3f3lzCa4JzttzEbCzcngw+WEemfeDw5oz/ULABKCSS5HQa83IiuVUqU4UnKaYrCeb5DqnCrJxQ/t2cnFdEYPSjE8bokurKiwzt2KtU6nRxJHLdz2vglu76KeKj0yVHfnN3vs4iKQZBRJr8lao/JqPdG1BaCtMR26iEBIkeH70/MiNm0pjXOB4dDo8RQP3AAfwUAL1PBZYJh1xcR/nb/h0+/pmeU/ZdW0xp+WpBDNHi89J7tLdYHdy11uq1SoudBMsRnDDv7LzL9InVGqnhCePTbJwj00UXLyZXEUf5+dgfp7gQo3DMMMmDfS2KN7PPl5+iK4mH94BoektQsNYbRO9Pr98E1m3J/lLSXEicr6WQuhhQl6SzP67HNOro2Wz72pyff3X2fwN7O6hPpp+gSAkGi2kyPa7miSkmphnYJJkHWCdzB79HlRV2LU5QJoBpCus1L/j+Q9SauQvjpGfHPYO/3BvSG29XE8n87N30evJ9RQA7euiDmnQ/9nkumVAaxKf4IZkRdetlQCGruYdP306HI3Auu65MS/BP2Je0kqpvR3UoqC5YeSgG4OGBAcFhZyFcqHZokob6r8Mky1e3aGyTeKyVuXUikOj2iS2BZQG2lo4W+I8Nxn8b9UP5KxrAcKQxDUijsT53H5+hg1XwIkuBU+oRGZliJhGtww4xBSZ5leyBOoSitcII0UkKzQynbBEIg/QeY70kqJFqUtJh/DOVLOZfjGZimkOO9WKJmghJISvbV9QAZ5U6MD5753l4dvjA3RJ3Oq7xuiP2Vp95pEZuOmlULqZNgPnlyenNunYh828t+a6RLv6kfs4VMWl25XqRTih30c/Cl5mVLmR+b+xE+9FmZuG391ZntvoqZ519FA4X4o8g35kbFRhLgJgIbW5xtwHcntnN+4tij317GX9Pe3VRqgqSD3c7zRUGbt+fO/Iqjp1HfD177au332gu2zrsObuejyyzjraKpBbN41HAvgtbXPZ2xAUUE/gPpOYLFvPVhswS4Ruql0pIU/1llrbOrukPAvylATOaaBuZHf7T9NFOh4H76YX76P59GoGxfvioNfu5wq40FZ9xqYvqO+0O02e+RmRqkg4sRo4aWsAelAQ7LDqlg+b/kuMt0VNJV5Ai7Ylai1ptRakTC/L2MSJkd2HvAwlZ0ewycXF7KeoWzx3OYzM5k1UgJ/DOBJFS1rTu4rcyfTseHiCBvB/2rKqwazAaUyXxLlY1/PgG0brAGQUnASjRks3pooC/0GttJSLGPPWRHNhbs0YRaUMrhPrjWRNvLcsYNDMXBavAfl1DfPQhq799cF8bvJicPuyiBImX+7W9v9tuLiDep7HFuhn6IkgFDZn99CnP5kKkwNSSpYC9V7bNagltmLRL6ADNQT4xRp8CAnJ4DDM0YJxqmzFsV/RzFoQBEYxpIDmC7XO2P/k9FDfujrRtNLH18XhUdfudOnVpog8HaIMr4zuHD4kFo2GVa3PbAWokV9s260R7Vt9NtzS6cOgaVtu6/QF8zySoBB9951zR1DXCL3o/uDreQp6JZ+hnupvSnj/0d/r93/2e9/uS3ALW1P7AVv9/v+CbTvJA8Ctia+Ost1veH00MWkX2ZqEcJ64dK4QltR+TTCtsbn4mU769dl7KEbatAUczc7OmzRvOmfMpJ8CXNiQiYRuGaOjtvUFYc2lOyBZUF0Lx0+Pjo7adTK0deEhlbKHfOPo3XAfEty/ABmbejWaGQAA" | base64 -d | gunzip | bash

    4.3 The cm-mission-control-setup setup wizard must be invoked with the --skip-packages option (required in air-gapped environments):

    cm-mission-control-setup --skip-packages

    4.4 Helm chart repository is not used in air-gapped mode if AJR installation. Enter dummy values for Helm repository username and password if asked.

    4.5 During the installation wizard, the air-gapped flow differs from the connected guide on the following screens:

    When prompted whether the Grafana public package repository must be configured, select no. In air-gapped mode this repository is not reachable.

    Prompt to configure Grafana repository.

    The wizard then prompts for the location of the Grafana Alloy packages downloaded by the nmc-airgap tool. Enter the path to the packages in the “Path” line or select a folder using the directory picker and click OK. The wizard verifies that the required packages are present in the specified directory before proceeding. The Alloy packages are installed into the necessary images during the deployment stage later.

    Prompt to select Alloy packages location.

    On the final screen, you must select Save config & deploy for the Alloy packages to be deployed during the air-gapped installation.

Once the installation is finished, follow the instructions in the Autonomous Job Recovery post-installation steps section for the post-installation steps and verification steps.