Air-Gapped Environment Deployment#

Overview#

Finetuning Microservices (FTMS) supports deployment in air-gapped environments, where Internet connectivity is restricted or unavailable. This feature enables organizations to use FTMS in secure, isolated environments while maintaining full functionality for model training, fine-tuning, and deployment.

The air-gapped deployment solution consists of three main components:

  1. Asset preparation (tao-core): Download and prepare pre-trained models, base experiments, and datasets in an Internet-connected environment

  2. Secure transfer: Transfer prepared assets (models + data) to the air-gapped environment via secure methods

  3. Chart deployment (tao-toolkit-api): Deploy FTMS in the air-gapped environment using Helm charts with SeaweedFS for distributed storage

Warning

Deployment in air-gapped environments requires disabling NVIDIA user authentication. Access control to air-gapped deployment must be managed by the user.

Key Features#

  • Complete asset download: Download entire pre-trained models for offline use

  • Flexible model selection: Support for auto-discovery, CSV-based, and architecture-specific selection modes

  • Distributed storage: Integration with SeaweedFS for scalable file storage

  • Secure transfer: Support for multiple secure transfer methods

Prerequisites#

Internet-connected environment (for preparation):

  • Access to NGC (NVIDIA GPU Cloud) with valid API keys

  • Python environment with TAO Core repository cloned

  • Sufficient storage space for model downloads (can be several gigabytes per model)

  • Sufficient storage space for datasets (can be several gigabytes to terabytes depending on data size)

Air-gapped environment (for deployment):

  • Kubernetes cluster

  • Helm 3.x installed

  • Sufficient storage for models and data

  • Storage class for persistent volumes

Architecture#

../../_images/airgapped-architecture.svg

Configuration#

Environment variables:

The following environment variables control air-gapped mode behavior:

# Enable air-gapped mode
export AIRGAPPED_MODE=true

# NGC API key for model downloads (preparation phase only)
export PTM_API_KEY="your_ngc_api_key"

Helm chart configuration:

Configure your values.yaml file for air-gapped deployment:

# Environment configuration
airgapped:
  enabled: true
  environment:
    AIRGAPPED_MODE: "true"
    LOCAL_MODEL_REGISTRY: "/shared-storage/ptm/airgapped-models"  # Must match SeaweedFS upload path

Note

If you are using Docker Compose: * You can skip the helm chart configuration and use the ./run.sh up-all --airgapped command to start the services in airgapped mode. * You should change the local model registry in config.env to match the SeaweedFS upload path. * You should update the secrets.json file with your PTM_API_KEY. * Refer to Docker Compose Deployment for more details.

Asset Preparation (Internet-Connected Environment)#

The asset preparation phase must be completed in an Internet-connected environment, and includes model preparation using the pretrained_models.py script from tao-core, dataset preparation, and container image management.

Airgapped Docker Images#

For airgapped deployments, you can use the existing Docker image save/load scripts to transfer TAO container images to your airgapped environment. This process involves saving images on an Internet-connected machine and loading them on the airgapped machine.

The key steps involve:

  1. Saving images on an Internet-connected machine using ./save-docker-images.sh

  2. Transferring the saved images to your airgapped environment

  3. Loading images on the airgapped machine using ./load-docker-images.sh

This approach ensures that all required TAO container images are available locally without requiring internet access during deployment.

Container Registry Setup#

  1. Login to NGC Private Registry.

    docker login nvcr.io

    Note

    You can use oauthtoken as a username and a valid NGC API key as a password to log in to the registry.

  2. Clone the FTMS “getting started” repository.

    git clone https://github.com/NVIDIA/tao_tutorials.git

  3. Navigate to the tao_tutorials/setup/tao-docker-compose repository.

    cd tao_tutorials/setup/tao-docker-compose

  4. On your Internet-connected machine, run the script to generate a folder named saved_docker_images in the root of the repository.

    ./save-docker-images.sh

  5. Use scp or rsync to copy the folder to your airgapped machine.

    scp -r saved_docker_images user@airgapped-machine:/path/to/saved_docker_images

  6. On the airgapped machine, run the script to load the container images into the local Docker registry.

    ./load-docker-images.sh

  7. Verify that the container images are loaded successfully into Docker.

    docker images

Model Preparation#

Use the pretrained_models.py script from tao-core to download pre-trained models.

Step 1. Set Up the Environment#

# Install tao-core package
pip install -U nvidia-tao-core

# Set environment variables
export AIRGAPPED_MODE=True
export PTM_API_KEY="your_ngc_api_key"

Step 2. Choose a Download Method#

The script supports multiple model discovery and download methods. Option 2 (CSV file mode) is recommended for getting started quickly as it allows you to select only the specific models needed for your workflows, while Option 1 downloads all available TAO models.

Option 1. Auto-Discovery Mode (All TAO Models)#

Automatically discovers and downloads all available models from specified organizations:

# Download all models from nvidia/tao organization
python -m nvidia_tao_core.microservices.pretrained_models \
  --org-teams "nvidia/tao" \
  --ngc-key "your_personal_ngc_key" \
  --shared-folder-path ./airgapped-models

# Auto-discover all accessible organizations
python -m nvidia_tao_core.microservices.pretrained_models \
  --ngc-key "your_personal_ngc_key" \
  --shared-folder-path ./airgapped-models
Option 3. Model Names Mode (By Architecture)#

Downloads models by matching specific architecture names.

python -m nvidia_tao_core.microservices.pretrained_models \
  --model-names "classification_pyt,dino,segformer,centerpose" \
  --ngc-key "your_personal_ngc_key" \
  --shared-folder-path ./airgapped-models
Option 4. Combined Mode#

Uses both CSV file and auto-discovery together.

python -m nvidia_tao_core.microservices.pretrained_models \
  --use-both \
  --org-teams "nvidia/tao" \
  --ngc-key "your_personal_ngc_key" \
  --shared-folder-path ./airgapped-models

Step 3. Verify Download#

# Verify download completed successfully
ls -la airgapped-models/
# Should contain: ptm_metadatas.json and model directories (nvidia/, nvstaging/, etc.)

Dataset Preparation#

Based on the TAO workflows you plan to run in an air-gapped environment, download and prepare the datasets in the Internet-connected environment:

# Create dataset directory structure
mkdir -p airgapped-datasets/{train,val,test}

# Copy your datasets to the prepared structure
cp -r /path/to/your/datasets/* airgapped-datasets/

Step 4. Package and Transfer Assets#

# Combine models and datasets into a single transfer package
mkdir -p airgapped-assets
mv airgapped-models airgapped-assets/models
mv airgapped-datasets airgapped-assets/datasets  # If you have datasets

# Create compressed archive and checksum
tar -czf airgapped-assets.tar.gz airgapped-assets/
sha256sum airgapped-assets.tar.gz > airgapped-assets.tar.gz.sha256

# Transfer to air-gapped environment (USB, SCP, etc.)
# cp airgapped-assets.tar.gz* /media/usb-drive/

Deployment (Air-Gapped Environment)#

Step 1. Extract Assets#

# Verify transfer integrity
sha256sum -c airgapped-assets.tar.gz.sha256

# Extract assets in air-gapped environment
tar -xzf airgapped-assets.tar.gz

# Verify extracted structure
ls -la airgapped-assets/
# Should contain: models/ and datasets/ directories

Step 2. Deploy FTMS and Set Up SeaweedFS#

# Deploy with air-gapped configuration
helm install tao-api /path/to/chart -f values.yaml

# Wait for pods to be ready
kubectl wait --for=condition=ready pod -l name=tao-api-app-pod --timeout=3600s

# Setup SeaweedFS access
CLUSTER_IP=$(kubectl get nodes -o jsonpath='{.items[0].status.addresses[?(@.type=="InternalIP")].address}')
SEAWEED_ENDPOINT=http://$CLUSTER_IP:32333
export AWS_ACCESS_KEY_ID=seaweedfs
export AWS_SECRET_ACCESS_KEY=seaweedfs123

Note

If you are using Docker Compose, you can set the SEAWEED_ENDPOINT environment variable to http://localhost:8333.

Step 3. Upload Assets to SeaweedFS#

Important

The SeaweedFS upload path for models must exactly match the LOCAL_MODEL_REGISTRY path in your values.yaml file.

# Create storage bucket and upload assets
aws s3 mb --endpoint-url $SEAWEED_ENDPOINT s3://tao-storage

# Upload models (path must match LOCAL_MODEL_REGISTRY)
aws s3 cp --endpoint-url $SEAWEED_ENDPOINT \
  airgapped-assets/models/ \
  s3://tao-storage/shared-storage/ptm/airgapped-models/ \
  --recursive

# Upload datasets (if any)
aws s3 cp --endpoint-url $SEAWEED_ENDPOINT \
  airgapped-assets/datasets/ \
  s3://tao-storage/data/ \
  --recursive

Step 4. Load Model Metadata#

Uses the FTMS API to load model metadata into the database.

import requests
import json

# Ensure that the airgapped-models/ directory with ptm_metadatas.json is uploaded.

# 1. Log in to TAO-API Service.
data = json.dumps({"ngc_org_name": ngc_org_name, "ngc_key": ngc_key})
response = requests.post(f"{host_url}/api/v1/login", data=data)
token = response.json()["token"]
base_url = f"{host_url}/api/v1/orgs/{ngc_org_name}"
headers = {"Authorization": f"Bearer {token}"}

# 5. Create a cloud workspace using the following metadata:
cloud_metadata = {
    "name": "SeaweedFS Workspace",
    "cloud_type": "seaweedfs",
    "cloud_specific_details": {
        "cloud_region": "us-east-1",
        "cloud_bucket_name": "tao-storage",
        "access_key": "seaweedfs",
        "secret_key": "seaweedfs123",
        "endpoint_url": "http://seaweedfs-s3:8333"
    }
}
endpoint = f"{base_url}/workspaces"
data = json.dumps(cloud_metadata)
response = requests.post(endpoint, headers=headers, data=data)
workspace_id = response.json()["id"]

# 6. Load base experiments into DB from below endpoint.
endpoint = f"{base_url}/experiments:load_airgapped"
data = {
    "workspace_id": workspace_id,
}
response = requests.post(endpoint, headers=headers, json=data)
# 2. Load base experiments into database
endpoint = f"{base_url}/experiments:load_airgapped"
data = {"workspace_id": workspace_id}
response = requests.post(endpoint, headers=headers, json=data)

Troubleshooting#

These are some common issues you may encounter:

  1. “Failed to download JSON file from cloud storage”: - Ensure that the SeaweedFS upload path matches LOCAL_MODEL_REGISTRY in values.yaml. - Verify that ptm_metadatas.json exists in uploaded models.

  2. SeaweedFS connection issues: - Check pods: kubectl get pods -l app.kubernetes.io/name=seaweedfs - Test connectivity: curl http://seaweedfs-s3:8333

  3. Path mismatch: - Verify that the models were uploaded to the correct S3 path matching LOCAL_MODEL_REGISTRY. - Use aws s3 ls to verify path structure.

Conclusion#

The FTMS air-gapped deployment feature enables you to create secure, isolated environments while maintaining full functionality. By combining model preparation from tao-core with the deployment infrastructure from tao-toolkit-api, you can meet strict security requirements while supporting complete ML/AI workflows.

On this page