> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/ccluster/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/ccluster/_mcp/server.

# gRPC Inference Endpoints

> Deploy and call a gRPC inference endpoint on CCluster, using vLLM as a worked example

General Inference can serve your container over **HTTP** (the default) or **gRPC**. gRPC runs over HTTP/2 and exchanges binary [Protocol Buffers](https://protobuf.dev/), which makes it a good fit for low-latency, high-throughput, or streaming inference workloads. We use **vLLM** as the example throughout this guide, deploying its gRPC server end to end.

## How gRPC works on CCluster

When you select **gRPC** as the protocol, a few things differ from an HTTP deployment:

* **Container port** — set it to the port your container serves gRPC on. This guide uses `50051` for vLLM.
* **Health checks** — CCluster verifies readiness with a TCP socket check on the container port, so the **Health check path** field is ignored.
* **External access** — like every deployment, the endpoint is exposed on **port 443 with TLS enforced**. CCluster's ingress terminates TLS on 443 and forwards traffic to your container's internal port. Clients connect to `<endpoint_url>:443`.

> **Note**
>
> The **container port** (e.g. `50051`) is used only inside the cluster. Externally your endpoint always answers on `443`, so gRPC clients should target `<endpoint_url>:443`, not the container port.

## 1. Configure the deployment

From the **General Inference** deployment page, fill in the basic details:

* **Add or Select Image** — `vllm/vllm-openai`.
* **Container Tag** — `latest`.
* **Container Port** — `50051` (the port the startup script below passes to `--port`).
* **Protocol** — select `gRPC`.
* **Hugging Face Token** — include one if your model is gated.

![General Inference form configured for a vLLM gRPC deployment](/ccluster/_fern-files/_img/8d6ccecacd9dedf8936bb9308721c3e7610324b169d3eff88713bde4f62224f2.webp)

> **Tip**
>
> This guide uses the pre-built `vllm/vllm-openai` image. To package and deploy your own gRPC container, see [Deploying Custom Models](/ccluster/resources/custom-image).

### Use a config-file startup script

The `vllm/vllm-openai` image does **not** bundle the gRPC extra, so startup is two steps: install `vllm[grpc]`, then launch the gRPC server. Put both in a startup script defined as a **Config File** and invoke it from **Add command**.

Keeping **Add command** to a simple script invocation is also the recommended pattern — commands with special characters (`.`, `,`, `[]`, and the like) placed directly in **Add command** can currently fail to deploy. This is a known issue CCluster is addressing.

Under **Optional Details → Config File**:

* **Filename** — `entry.sh`
* **Mount Path (directory)** — `/etc/start`
* **File Content**:

```bash showLineNumbers={false}
python3 -m pip install -q --root-user-action=ignore vllm[grpc]
vllm serve meta-llama/Llama-2-7b-hf --grpc --port 50051
```

`--port 50051` is required: in gRPC mode `vllm serve` binds its regular `--port`, which defaults to `8000`.

> **Warning**
>
> **Deprecated:** `python3 -m vllm.entrypoints.grpc_server --model <model>` is deprecated as of vLLM 0.30 and may be removed in a future release. Use `vllm serve <model> --grpc` instead.

The file is mounted at `<mount path>/<filename>`, so set **Add command** to run it:

```bash showLineNumbers={false}
/bin/sh /etc/start/entry.sh
```

![Config File entry.sh installing vllm\[grpc\] and launching the gRPC server, invoked from Add command](/ccluster/_fern-files/_img/f0885eb16494aa411ec7eb35e406af549feaed1c31924e6828cbe9a1fc388827.webp)

> **Tip**
>
> Every endpoint is protected — there are no public endpoints. Secure access with a Bearer token, an mTLS client certificate, or both. For mTLS, select **Make it a private endpoint?** and CCluster generates a TLS certificate that clients must present. See [Securing Endpoints](/ccluster/resources/private) for details.

## 2. Select the cluster and hardware

Choose the cluster and GPU instance that match your model and setup, then click **Deploy**.

![Hardware instance selection](/ccluster/_fern-files/_img/7293986fc2b6e2e43f89b126c57661ab033825c6d9d56c54b8717262c4836fa1.webp)

## 3. Call your gRPC endpoint

Once the deployment is **Ready**, its endpoint URL appears on the deployment's **Detail** page under **Endpoint Configuration**, alongside the protocol (gRPC). Clients connect to this host at `<endpoint_url>:443` over TLS. Because CCluster terminates TLS on 443, point your gRPC client there with TLS enabled (for `grpcurl`, that means omitting `-plaintext`).

vLLM's gRPC server has reflection enabled, so you can list its services with [`grpcurl`](https://github.com/fullstorydev/grpcurl):

```bash showLineNumbers={false}
grpcurl <endpoint_url>:443 list
```

```
grpc.health.v1.Health
grpc.reflection.v1alpha.ServerReflection
vllm.grpc.engine.VllmEngine
```

Check readiness with the standard gRPC health service:

```bash showLineNumbers={false}
grpcurl <endpoint_url>:443 grpc.health.v1.Health/Check
# {"status": "SERVING"}
```

Inspect the served model:

```bash showLineNumbers={false}
grpcurl -d '{}' <endpoint_url>:443 vllm.grpc.engine.VllmEngine/GetModelInfo
```

Run inference with the server-streaming `Generate` RPC. The response streams `token_ids` rather than decoded text — decode them with the tokenizer (the `GetTokenizer` RPC) or see the [vLLM documentation](https://docs.vllm.ai/) for the full request and response schema:

```bash showLineNumbers={false}
grpcurl -d '{"request_id": "1", "text": "The capital of France is", "sampling_params": {"temperature": 0.7, "max_tokens": 16}, "stream": true}' \
  <endpoint_url>:443 vllm.grpc.engine.VllmEngine/Generate
```

> **Note**
>
> Endpoints are protected, so authenticate every call — pass a Bearer token, or for an mTLS endpoint pass the downloaded certificate to your client (for `grpcurl`, use `-cert` / `-key`). See [Securing Endpoints](/ccluster/resources/private).

## What's next

#### [Deploying Custom Models](/ccluster/resources/custom-image)

Build your own containerized gRPC inference engine and deploy it.

#### [Clients](/ccluster/clients/setup)

Interact with NVIDIA CCluster programmatically.

#### [Managing Vault Objects](/ccluster/resources/vault)

Store the Hugging Face tokens and registry credentials your deployments use.