> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/ccluster/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/ccluster/_mcp/server.

# Deploying NVIDIA NIM

> Deploy NVIDIA NIM microservices on CCluster through General Inference

[NVIDIA NIM](https://developer.nvidia.com/nim) microservices are self-contained containers that bundle a model-optimized inference engine and download model weights automatically at startup, exposing an OpenAI-compatible API. NIM is not a pre-configured recipe on the [General Inference](/ccluster/apps/inference) page — enter the image manually with the settings below.

## Find a NIM image

NIM images are hosted on `nvcr.io`. Browse the catalog at [build.nvidia.com](https://build.nvidia.com/explore/discover) and copy the image path from the model's **Deploy** tab. The catalog shows the image and tag joined with a colon (e.g. `nvcr.io/nim/meta/llama-3.1-8b-instruct:latest`) — on CCluster, split them into the **Add or Select Image** and **Container Tag** fields:

| Image                                                  | Tag      |
| ------------------------------------------------------ | -------- |
| `nvcr.io/nim/deepseek-ai/deepseek-r1-distill-qwen-32b` | `latest` |
| `nvcr.io/nim/meta/llama-3.1-8b-instruct`               | `latest` |

You also need an [NGC API Key](https://ngc.nvidia.com) — it is used both to pull the image and to download model weights.

## Configure the deployment

From the **General Inference** deployment page, every NIM uses the same configuration:

| Field                       | Value                                                                |
| --------------------------- | -------------------------------------------------------------------- |
| **Add or Select Image**     | The `nvcr.io/nim/...` image path (entered manually, without the tag) |
| **Container Tag**           | The image tag (e.g. `latest`)                                        |
| **Container Port**          | `8000`                                                               |
| **Protocol**                | `HTTP`                                                               |
| **Health check path**       | `/v1/health/ready`                                                   |
| **Image Registry Username** | `$oauthtoken` (literal string — not a variable)                      |
| **Image Registry Password** | Your [NGC API Key](https://ngc.nvidia.com)                           |

Leave **Add command** empty — the image's default entrypoint starts the server. Under **Optional Details → Environment Variables**, also add:

| Key           | Value            |
| ------------- | ---------------- |
| `NGC_API_KEY` | Your NGC API Key |

The NGC API Key is needed twice: the registry credentials let CCluster pull the image from `nvcr.io`, and the `NGC_API_KEY` environment variable lets the container download model weights on first boot. Without the environment variable, the container starts but fails to load the model.

> **Note**
>
> NIM downloads model weights at startup, so the first boot can take several minutes for large models. The deployment stays in a pending state until the `/v1/health/ready` probe passes.

From here the deployment follows the standard General Inference flow — select the cluster and hardware, click **Deploy**, and monitor the deployment until it is ready. See [General Inference](/ccluster/apps/inference) for details. Check the model's page on [build.nvidia.com](https://build.nvidia.com/explore/discover) for the GPU and memory each NIM supports.

Once ready, query the endpoint like any OpenAI-compatible endpoint. List the served model identifier first, then send requests:

```bash showLineNumbers={false}
# Find the model identifier served by this NIM
curl https://<endpoint_url>/v1/models \
  -H "Authorization: Bearer <your_bearer_token>"

# Chat completion
curl https://<endpoint_url>/v1/chat/completions \
  -H "Authorization: Bearer <your_bearer_token>" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta/llama-3.1-8b-instruct",
    "messages": [{"role": "user", "content": "Hello"}],
    "max_tokens": 256
  }'
```

> **Tip**
>
> Every endpoint is protected — there are no public endpoints. Secure access with a Bearer token, an mTLS client certificate, or both. For mTLS, select **Make it a private endpoint?** and CCluster generates a TLS certificate that clients must present. See [Securing Endpoints](/ccluster/resources/private) for details.

## What's next

#### [General Inference](/ccluster/apps/inference)

Deploy and scale any containerized model on CCluster.

#### [Securing Endpoints](/ccluster/resources/private)

Protect your endpoints with Bearer tokens and mTLS certificates.

#### [Managing Vault Objects](/ccluster/resources/vault)

Store the registry credentials and API keys your deployments use.