Skip to navigation
ReferenceProjects and Environments

Multi-Container Environments (Docker Compose)

View as Markdown

This page documents the technical specifications for using Docker Compose with AI Workbench projects.

Requirements

  • AI Workbench project with Docker Compose file in the repository.
  • Basic understanding of Docker Compose syntax and concepts.
  • For GPU services: NVIDIA GPU available on the host.

For more information about Docker Compose, see the Docker Compose documentation.

Compose File Specifications

The compose file is a YAML file that defines the services, networks, and volumes for a multi-container environment.

File Names and Locations

The compose file has restrictions in regards to the file name and location.

Valid File Names

AI Workbench recognizes the following file names:

  • compose.yaml
  • compose.yml
  • docker-compose.yml
  • docker-compose.yaml

Default Locations

AI Workbench expects the compose file to be in one of the following locations:

  • Project root: /project/
  • Deploy folder: /project/deploy/

Custom Location

You can specify a different location by editing the project specification file:

  1. Add the field environment:compose-file-path to .project/spec.yaml
  2. Set the value to the relative path to the compose file in the project

For more information about the project specification file, see AI Workbench Project Specification.

Service Profiles

Service profiles allow conditional service activation based on hardware or environment requirements.

For complete information about service profiles, see the Docker Compose profiles documentation.

Usage in AI Workbench:

  • Select one or more profiles in Project Tab > Environment > Compose before starting.
  • Services without a profile always run.
  • Services with a profile only run when that profile is selected.

GPU Configuration in Compose

GPU configuration in compose specifies GPU allocation for services requiring GPU acceleration.

GPU Allocation by Count

You can specify GPU requests using the deploy.resources.reservations format with a count value.

AI Workbench manages GPU reservations and explicitly passes GPUs into each container, preventing conflicts between services.

Example with count:

services:
  gpu-service:
    image: nvidia/cuda:12.2.0-base-ubuntu22.04
    runtime: nvidia
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

Configuration Fields:

  • runtime: Should be nvidia for GPU access
  • driver: Must be nvidia
  • count: Number of GPUs to allocate to this service (Docker assigns available GPUs)
  • capabilities: Must include [gpu]

GPU Allocation by Device ID

You can assign specific GPU devices to services using device_ids instead of count.

Example with device_ids:

services:
  embedding-service:
    image: nvcr.io/nim/nvidia/llama-3.2-nv-embedqa-1b-v2:latest
    runtime: nvidia
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              device_ids: ['0']
              capabilities: [gpu]
  llm-service:
    image: nvcr.io/nim/meta/llama-3.1-70b-instruct:latest
    runtime: nvidia
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              device_ids: ['1', '2']
              capabilities: [gpu]

Configuration Fields:

  • device_ids: List of specific GPU device IDs to assign
  • Device IDs correspond to GPU indices shown in nvidia-smi output
  • Multiple device IDs can be specified for multi-GPU services

When to use device_ids vs count:

  • Use count when you don’t care which specific GPUs are assigned
  • Use device_ids when you need precise GPU assignment across multiple services
  • Use device_ids to prevent services from competing for the same GPU

Multi-GPU Configurations

Services requiring multiple GPUs can specify them using either count or device_ids.

Example with multiple GPUs:

services:
  large-model:
    image: nvcr.io/nim/meta/llama-3.1-405b-instruct:latest
    runtime: nvidia
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 8
              capabilities: [gpu]
    shm_size: 20gb

Shared Memory for Multi-GPU:

  • Multi-GPU services often require increased shared memory
  • Set shm_size to allocate shared memory (e.g., 20gb, 16GB)
  • Required for inter-GPU communication in many AI workloads

GPU Allocation Conflicts

AI Workbench handles GPU reservation conflicts - if requested GPUs exceed available GPUs, the compose environment will fail to start with an error message.

Common conflict scenarios:

  • Project container using GPUs that compose services request
  • Multiple services requesting the same specific device_ids
  • Total count across all services exceeds available GPUs

Resolution:

  • Check GPU availability with nvidia-smi before starting services
  • Ensure the project container is not using GPUs needed by compose services
  • Use device_ids to explicitly control GPU assignment across services

CPU and Memory Resources

Services can specify CPU and memory resource limits and reservations to prevent resource exhaustion.

Resource Limits

Resource limits define the maximum resources a service can consume.

Example:

services:
  my-service:
    image: myapp/service:latest
    deploy:
      resources:
        limits:
          cpus: '2.0'
          memory: 4G
        reservations:
          cpus: '1.0'
          memory: 2G

Configuration Fields:

  • limits.cpus: Maximum CPU cores (e.g., '2.0' for 2 cores, '0.5' for half a core)
  • limits.memory: Maximum memory (e.g., 4G, 512M)
  • reservations.cpus: Minimum CPU cores reserved for the service
  • reservations.memory: Minimum memory reserved for the service

Best Practices:

  • Set limits for inference services to prevent consuming all host resources
  • Use reservations to guarantee minimum resources for critical services
  • Important when running multiple AI services on a single host

Environment Variables and Secrets

You can use environment variables and secrets in your compose file to configure your services.

Environment Variables

Environment variables defined in your AI Workbench project are available in compose services through interpolation:

services:
  myservice:
    environment:
      - MY_VAR=${TEST_VAR}

Special Environment Variables

  • PROXY_PREFIX: Automatically injected into services, contains the proxy routing path.
  • NVWB_TRIM_PREFIX: Set to true to enable proxy routing with prefix trimming.

Secrets

Secrets are used to store sensitive information in your compose file.

You can use AI Workbench secrets in your compose file in two ways:

  1. Interpolation (as environment variables):

    environment:
      - MY_SECRET=${SECRET_NAME}

  2. Compose secrets (as files):

    services:
      myservice:
        secrets:
          - SECRET_NAME
        environment:
          - SECRET_FILE=/run/secrets/SECRET_NAME
    secrets:
      SECRET_NAME:
        environment: "HOME"

When using compose secrets, the global secrets section must include the secret name with environment: "HOME". AI Workbench automatically replaces this value at runtime.

Networks

Networks enable service-to-service communication in multi-container environments.

Network Configuration

Services on the same network can reach each other using service names as hostnames.

Example:

services:
  frontend:
    image: myapp/frontend:latest
    networks:
      - app-network
    environment:
      - API_URL=http://backend:8080
  backend:
    image: myapp/backend:latest
    networks:
      - app-network
    ports:
      - "8080:8080"
networks:
  app-network:
    driver: bridge

Network Types:

  • bridge: Default network driver for standalone containers on a single host
  • Services on the same bridge network can communicate by service name

Service Discovery:

  • A service named backend is accessible at hostname backend from other services
  • Port numbers in inter-service communication refer to container ports, not host ports
  • External port mappings (ports) expose services to the host or AI Workbench proxy

Multiple Networks:

Services can connect to multiple networks for network segmentation and isolation.

services:
  app:
    networks:
      - frontend-network
      - backend-network
networks:
  frontend-network:
  backend-network:

Mounts and Volumes

You can use mounts and volumes in your compose file to share files and data between your services.

Volume Types

Docker Compose supports three volume types for persistent storage:

Named Volumes:

Managed by the container engine (Docker or Podman) and are useful for data that should persist but doesn’t need direct host access.

services:
  database:
    volumes:
      - db-data:/var/lib/postgresql/data
volumes:
  db-data:

Bind Mounts:

Connect to specific host directories, useful for development or accessing existing data.

services:
  nim:
    volumes:
      - type: bind
        source: /tmp
        target: /opt/nim/.cache/

Tmpfs Mounts:

Store data in host memory, useful for temporary sensitive data.

services:
  app:
    tmpfs:
      - /tmp

Shared Volume

AI Workbench automatically creates a shared volume accessible to all containers. This shared volume has the following characteristics:

  • Mount path: /nvwb-shared-volume
  • Purpose: Enables file sharing between the project container and compose services.
  • Permissions: If services run as different users, you may need to modify file permissions for shared access.

Bind Mounts

AI Workbench does not manage bind mount source paths for compose containers.

You must specify bind mount source paths directly in the compose file. These paths are relative to the project or absolute paths on the host.

Common Bind Mount Patterns:

services:
  service:
    volumes:
      # Relative to project root
      - ./data:/app/data
      # Absolute host path
      - /mnt/datasets:/data
      # Model cache directory
      - ${MODEL_DIRECTORY:-/tmp}:/opt/nim/.cache

External Volumes

External volumes are managed outside of Compose and must be created before starting services.

services:
  nim:
    volumes:
      - nim_cache:/opt/nim/.cache
volumes:
  nim_cache:
    external: true

Create external volumes with docker volume create nim_cache before running compose.

Service Dependencies and Healthchecks

Service dependencies control startup order and ensure services are ready before dependent services start.

depends_on Configuration

The depends_on option specifies service dependencies.

Basic Dependency:

services:
  frontend:
    depends_on:
      - backend
  backend:
    image: myapp/backend:latest

With Healthcheck Conditions:

services:
  frontend:
    depends_on:
      backend:
        condition: service_healthy
  backend:
    image: myapp/backend:latest
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8080/health"]
      interval: 10s
      timeout: 5s
      retries: 3

Dependency Conditions:

  • service_started: Default, waits for service to start (not necessarily ready)
  • service_healthy: Waits for service to pass healthcheck
  • service_completed_successfully: Waits for service to complete and exit successfully

Healthcheck Configuration

Healthchecks verify that a service is ready to accept requests.

Healthcheck Fields:

services:
  nim-llm:
    healthcheck:
      test: ["CMD", "python3", "-c", "import requests; requests.get('http://localhost:8000/v1/health/ready')"]
      interval: 10s
      timeout: 20s
      retries: 100
      start_period: 10m

Configuration Options:

  • test: Command to run (returns 0 for healthy, 1 for unhealthy)
  • interval: Time between healthcheck attempts
  • timeout: Maximum time to wait for healthcheck to complete
  • retries: Number of consecutive failures before marking unhealthy
  • start_period: Grace period before first healthcheck (useful for slow-starting services)

Common Healthcheck Patterns:

HTTP endpoint check:

test: ["CMD", "curl", "-f", "http://localhost:8000/health"]

Python script check:

test: ["CMD", "python3", "-c", "import requests; requests.get('http://localhost:8000/v1/health/ready')"]

Port check:

test: ["CMD-SHELL", "nc -z localhost 8080"]

Best Practices:

  • Define healthchecks for all critical services
  • Use appropriate start_period for services that take time to initialize
  • Set realistic retries and timeout values for AI services that may take time to respond
  • Healthchecks are essential for complex multi-service pipelines

Versioning

The compose file is versioned with the project. As long as the compose file is in a tracked folder in the project, Git will version it.

What is Versioned

Versioning occurs when the compose file is in a tracked folder in the project. As long as the compose file is in a tracked folder in the project, Git will version it.

What is NOT Versioned

The component images are not version when those images are used for services are not versioned. They are not versioned for the following reasons:

  • Service container images are pulled from registries at runtime.
  • Image versions are determined by tags in the compose file.
  • Updating the compose file and committing changes does not version the actual container images.

Examples

Example 1: Simple Web Service

A basic compose file with a single web service that always runs:

services:
   web1:
      image: hashicorp/http-echo
      environment:
         # NVWB_TRIM_PREFIX=true enables proxy routing with prefix trimming
         # PROXY_PREFIX is automatically injected
         - NVWB_TRIM_PREFIX=true
      ports:
         - '5678:5678'
      command: ["-text=hello from service 1"]

Example 2: GPU-Enabled Service with Profile

A compose file with two services - one always runs, one requires GPU and uses a profile:

services:
   web1:
      image: hashicorp/http-echo
      environment:
         - NVWB_TRIM_PREFIX=true
      ports:
         - '5678:5678'
      command: ["-text=hello from service 1"]
   web2:
      image: hashicorp/http-echo
      profiles: [gpu-service]
      environment:
         - NVWB_TRIM_PREFIX=true
      ports:
         - '5679:5679'
      # GPU allocation specification
      deploy:
        resources:
          reservations:
            devices:
              - driver: nvidia
                count: 1
                capabilities: [gpu]
      command: ["-text=hello from service 2", "-listen=:5679"]

Example 3: Environment Variables and Secrets via Interpolation

A compose file using AI Workbench environment variables and secrets through interpolation:

services:
   web3:
      image: hashicorp/http-echo
      environment:
         - NVWB_TRIM_PREFIX=true
         # Environment variables from AI Workbench project
         - TEST_ENV_VAR=${TEST_VAR}
         # Secrets also available via interpolation
         - TEST_SECRET_FROM_ENV_VAR=${TEST_SECRET}
      ports:
         - '5678:5678'
      command: ["-text=${TEST_VAR}"]

Create the variable TEST_VAR and secret TEST_SECRET in your AI Workbench project before using this example.

Example 4: Compose Secrets as Files

A compose file using AI Workbench secrets through the Compose secrets mechanism:

services:
   web1:
      image: hashicorp/http-echo
      environment:
         - NVWB_TRIM_PREFIX=true
      ports:
         - '5678:5678'
      command: ["-text=hello from service 1"]
   web4:
      image: hashicorp/http-echo
      profiles: [compose-secret]
      environment:
         - NVWB_TRIM_PREFIX=true
         # Secret mounted as file by compose
         - TEST_SECRET_FILE=/run/secrets/TEST_SECRET
      ports:
         - '5680:5680'
      # Link to the global secrets section
      secrets:
         - TEST_SECRET
      command: ["-text=hello from service 4", "-listen=:5680"]
# Global secrets section - AI Workbench replaces the value at runtime
# The name must match the secret name in your AI Workbench project
secrets:
  TEST_SECRET:
    environment: "HOME"

Create the secret TEST_SECRET in your AI Workbench project before using this example.