Examples#

These examples illustrate the designs in Architecture. They use generic hostnames and placeholders. They are configuration fragments for review, not a complete installation procedure. Select the examples that match your network policy, installed versions, registry, and workload manager.

The BCM examples target BCM 11. The Harbor example follows its 2.14 documentation, and the Zot example follows its 2.1.15 documentation. Containerd configuration paths and plugin names vary by version and BCM integration. Confirm the active configuration before applying a fragment. The examples have not undergone live workload validation.

Proxy Scopes and Internal Bypasses#

NVIDIA Base Command Manager (BCM) host settings, package-manager settings, container runtime settings, and workload settings serve different clients. A host shell environment does not configure every system service or container. The following table identifies the configuration scope for each client.

Client	Illustrative Configuration Scope	Internal Endpoints to Consider
BCM-managed host tools	BCM device proxy settings and the environment used by each tool.	BCM head nodes, loopback, internal services, and local registry hosts.
Advanced Package Tool (APT)	A separate APT configuration file on the head node and in software images.	Local package mirrors.
Containerd image pulls	Registry host mappings, registry trust, and the runtime service environment when proxy egress is permitted.	Local registry and authentication endpoints.
Registry upstream requests	Registry service proxy environment, upstream credentials, and trust.	Local registry services and storage.
Workload downloads	Explicit environment or application configuration inside the workload.	Kubernetes API and service names, model services, and local object storage.

You can include loopback, the head node, local registries, storage services, and Kubernetes API and service endpoints in each relevant bypass list. List required hostnames and addresses explicitly. Domain suffix and Classless Inter-Domain Routing (CIDR) matching differ between clients; do not assume that a CIDR accepted by BCM works in every application’s NO_PROXY parser. Check the short names, fully qualified names, and addresses that clients actually use.

BCM 11 Host Settings#

BCM exposes per-device proxy settings through the proxysettings submode. The following cmsh session illustrates HTTP and HTTPS proxy addresses plus direct access to internal hosts. Replace <bcm-device> with a managed Linux device, and adapt the bypass list to your site. Refer to the BCM 11 Administrator Manual, Section 3.10.5.

device
proxysettings <bcm-device>
set httpproxyaddress http://proxy.example.com:3128
set httpsproxyaddress http://proxy.example.com:3128
set hostswithoutproxy localhost,127.0.0.1,::1,bcm.example.com,registry.example.com,packages.example.com,api.example.com
commit
show

The http:// proxy URL is also appropriate for an HTTPS destination when Squid uses CONNECT. It names the connection to the proxy, not the destination protocol. Apply the policy to the head node and relevant regular nodes through your BCM management process. For a fully disconnected design, use local sources and remove obsolete outbound proxy settings through that process.

Separate APT Settings and Software-Image Persistence#

You can manage a separate file such as /etc/apt/apt.conf.d/90-site-proxy on the head node and in every active Ubuntu software image. The following fragment routes upstream requests through the proxy while bypassing a local package host. Refer to the APT HTTP transport and APT HTTPS transport references.

Acquire::http::Proxy "http://proxy.example.com:3128/";
Acquire::https::Proxy "http://proxy.example.com:3128/";
Acquire::http::Proxy::packages.example.com "DIRECT";
Acquire::https::Proxy::packages.example.com "DIRECT";

For a software image whose root is <software-image-root>, the corresponding file is <software-image-root>/etc/apt/apt.conf.d/90-site-proxy. Include the package source definitions and required CA certificates in the software image. A live-node edit alone can disappear after reprovisioning. You can use BCM’s supported software-image update workflow, including imageupdate where appropriate, to propagate the file to existing nodes. Review exclusions and schedule the update for the selected nodes before applying it.

In a fully disconnected environment, point APT sources at local repositories or the retained installation media instead of public repositories. Removing the proxy does not make a public repository available offline. Keep repository signature checks enabled.

The following read-only check shows the effective APT proxy values on a host. You can compare the head node, software image, and a reprovisioned node to confirm persistence.

apt-config dump | grep -E '^Acquire::(http|https)::Proxy'

Process and Workload Environment#

The following shell fragment illustrates proxy variables for a client that supports them. The lower-case names are common; some libraries require upper-case names as well. A system service needs its own managed environment. A Kubernetes workload needs the variables in its pod configuration, and a Slurm job needs them in its job environment.

export http_proxy='http://proxy.example.com:3128'
export https_proxy='http://proxy.example.com:3128'
export no_proxy='localhost,127.0.0.1,::1,bcm.example.com,registry.example.com,api.example.com,.svc,.cluster.local'
export HTTP_PROXY="$http_proxy"
export HTTPS_PROXY="$https_proxy"
export NO_PROXY="$no_proxy"

This fragment assumes the cluster uses cluster.local as its service domain. Replace it if your cluster uses another domain, and add the required local addresses. Test the application’s matching rules. These variables provide application download access; they do not redirect a Run:ai workload’s container image pull.

Local Registries and Cache Examples#

A registry can hold deliberately imported offline content, fetch upstream content on demand, or provide separate repositories for both roles. Choose a service that meets your authentication, retention, availability, and storage requirements. Harbor and Zot illustrate different configuration styles; neither is required by this architecture.

Harbor Proxy Project and Offline Project#

In Harbor, you can configure an upstream registry endpoint and select it for a project with Proxy Cache enabled. For a proxy project named cache-upstream, a repository named team/training becomes the following local pull reference.

docker pull 'harbor.example.com/cache-upstream/team/training:<tag>'

Preserve the complete upstream repository path. For a Docker Hub official image, that path includes library, such as cache-upstream/library/<image>:<tag>. A first pull needs upstream access. Local clients authenticate to Harbor; the registry endpoint separately authenticates to the upstream. Harbor does not accept pushes into a proxy project. Use an ordinary project such as approved for transferred offline images, with references such as harbor.example.com/approved/team/training:<tag>. Refer to Harbor proxy cache configuration.

Review the project’s retention policy before relying on cached content. Confirm that the registry service can reach the upstream through the approved proxy and that all participating clients trust the local certificate. A successful cached pull does not establish that a cache miss will succeed.

Zot On-Demand Synchronization#

Zot can synchronize images from an upstream OCI registry. The following fragment belongs under extensions in an existing Zot configuration. It illustrates on-demand fetching from a generic upstream. It omits the storage, listener, authentication, and certificate configuration required for a complete service.

Zot Synchronization Example#

{
  "sync": {
    "enable": true,
    "registries": [
      {
        "urls": ["https://upstream.example.com"],
        "onDemand": true,
        "tlsVerify": true
      }
    ]
  }
}

With on-demand synchronization, an absent image requires upstream access. Zot also supports scheduled synchronization for selected content. For a fully disconnected service, import and verify the approved content before use; disabling on-demand synchronization does not populate storage. Zot can convert Docker-format images to OCI format, which can change digests and affect signatures. Confirm the installed release’s compatibility and preserveDigest options before choosing a transfer method. Refer to Zot 2.1.15 mirroring.

Slurm, Pyxis, and Enroot#

Slurm with Pyxis uses Enroot for container imports. The following examples assume Pyxis and Enroot are already configured, the selected image matches the node architecture, and the user can read the registry and shared storage. Refer to Slurm Workload Management for the Slurm administration context.

Pulling from a Local Registry#

The following command illustrates a Pyxis image reference to a local registry. The # separates the registry hostname from the repository path in Enroot syntax. Quoting preserves that literal separator.

srun --container-image='registry.example.com#approved/team/training:<tag>' <command>

Enroot needs its own registry credentials and CA trust. Kubernetes pull secrets and containerd mirror settings do not configure it. Enroot documents credentials in $ENROOT_CONFIG_PATH/.credentials using the netrc format. Handle that file as a credential rather than placing it in shared examples. Refer to the NVIDIA Enroot import reference.

Using a Shared SquashFS File#

The following example imports an image into a SquashFS file, then uses that file for a Slurm job. It assumes the destination directory exists and is accessible to the importing user and compute nodes.

enroot import --output '/shared/containers/<image-version>.sqsh' 'docker://registry.example.com#approved/team/training:<tag>'
srun --container-image='/shared/containers/<image-version>.sqsh' <command>

You can also prepare a SquashFS file on a compatible preparation host and transfer it through the approved process. Record its checksum and architecture. The file avoids a registry pull for the job’s container filesystem, but application downloads still need local files or approved proxy access. Size shared storage for the retained files and nodes for Enroot’s extraction and runtime space. Refer to the NVIDIA Pyxis usage reference.

Run:ai and Containerd#

Run:ai workloads use Kubernetes image pulls on the scheduled node. You can use explicit local references such as registry.example.com/approved/team/training:<tag> in workload configuration. For a private repository, provide the supported workload or namespace registry credential configuration; Kubernetes imagePullSecrets must exist in the workload namespace. Refer to Kubernetes private registry authentication.

If you retain upstream image names, an approved containerd mapping can route their registry namespace to a local proxy project. The following hosts.toml fragment illustrates a local-only mapping for upstream.example.com. It uses the local service as server, so the upstream registry is not the fallback. The host entry makes the Harbor project path and CA explicit.

server = "https://harbor.example.com/v2/cache-upstream"
[host."https://harbor.example.com/v2/cache-upstream"]
  capabilities = ["pull", "resolve"]
  override_path = true
  ca = "/etc/ssl/certs/<registry-ca>.pem"

Place the namespace directory upstream.example.com under the runtime’s active registry config_path, and put hosts.toml inside it. BCM-managed containerd can use a path under /cm/local/apps/containerd/var/etc/certs.d/; do not assume that a file under /etc/containerd/certs.d/ is active. Confirm the runtime version, effective config_path, and BCM ownership before changing files, then persist the approved mapping and CA in the relevant software images. Refer to the containerd registry host reference.

A [host] mirror with an upstream server, or an omitted server, can allow upstream fallback. Review that behavior explicitly. The fragment does not configure private registry authentication or the registry service’s outbound proxy. A Container Runtime Interface (CRI) pull through the running service tests a different path from a standalone ctr command.

Application Downloads After Image Pull#

A locally available container image does not imply that its model, dataset, or Python packages are available. You can mount approved files from shared storage and configure the application’s documented local paths or offline mode. If restricted egress is permitted, pass the application’s proxy and bypass settings separately. Retain exact model revisions and all supporting files so that a startup script does not silently request a new upstream version.

For OCI charts, Helm supports registry login and versioned chart pulls. The following example assumes the local chart repository and its CA trust already exist. It fetches the chart, not the images referenced by the chart. Refer to the Helm OCI registry reference.

helm registry login registry.example.com
helm pull 'oci://registry.example.com/charts/<chart>' --version '<version>'

Suggested Validation and Failure Symptoms#

You can validate each client separately before a broader workload check. The following suggestions distinguish network access from artifact completeness.

Compare BCM proxy values and effective APT settings on the head node, in software images, and on a reprovisioned node.
Request the local registry’s /v2/ endpoint with normal certificate verification and an explicit internal bypass. A 200 response or an expected 401 authentication challenge confirms an API response, not a successful image pull.
Pull an approved image through Enroot and through the running containerd CRI service on the intended nodes. Use the configured workload credentials and verify the expected architecture and digest.
In a permitted test window, check a cached image and an approved cache miss separately. For an offline design, check pulls and application startup while upstream access is unavailable and the test node has no cached copy of the image.
Check that a representative workload opens its model and dataset from the intended local source. Inspect registry, runtime, application, and proxy logs to confirm which endpoints it contacts.
Restore representative retained artifacts to a separate service and check that consumers can use them.

The following table lists common symptoms and the configuration area you can inspect first.

Symptom	Suggested First Check
Timeout to an internal endpoint	Bypass matching, local name resolution, routing, and the client-specific proxy environment.
Proxy denial or `407` response	Squid destination policy or required proxy authentication.
Registry `401` or `403` during a pull	Local repository permissions, pull credentials, and separate upstream endpoint credentials.
`x509` certificate error	The CA trusted by the failing client, certificate chain, hostname, and expiry.
Repository or manifest not found	Local project prefix, complete repository path, retained tag or digest, and required architecture.
Cache hit succeeds but cache miss fails	Registry upstream reachability, proxy settings, authentication services, and redirects.
`ImagePullBackOff` or `ErrImagePull`	Pod events and the scheduled node’s runtime path, credentials, trust, and local artifact inventory.
Container starts but model loading fails	Local model path, complete revision, file permissions, and application download settings.
Settings disappear after reprovisioning	Software-image configuration, managed file exclusions, and BCM propagation.
Import fails with insufficient space	Registry capacity, Enroot cache and temporary storage, extraction space, and shared storage quotas.

These checks provide evidence for your site’s selected configuration. They do not establish production readiness for another version, consumer, or artifact set.