> For clean Markdown content of this page, append .md to this URL. For the complete documentation index, see https://docs.nvidia.com/dynamo/llms.txt. For full content including API reference and SDK examples, see https://docs.nvidia.com/dynamo/llms-full.txt.

# Frontend

The Dynamo Frontend is the API gateway for serving LLM inference requests. It provides OpenAI-compatible HTTP endpoints and KServe gRPC endpoints, handling request preprocessing, routing, and response formatting.

## Feature Matrix

| Feature | Status |
|---------|--------|
| OpenAI Chat Completions API (`/v1/chat/completions`) | ✅ Supported |
| OpenAI Completions API (`/v1/completions`) | ✅ Supported |
| OpenAI Embeddings API (`/v1/embeddings`) | ✅ Supported |
| OpenAI Responses API (`/v1/responses`) | ✅ Supported |
| OpenAI Models API (`/v1/models`) | ✅ Supported |
| Image Generation (`/v1/images/generations`) | ✅ Supported |
| Video Generation (`/v1/videos/generations`) | ✅ Supported |
| Anthropic Messages API (`/v1/messages`) | 🧪 Experimental |
| KServe gRPC v2 API | ✅ Supported |
| Streaming responses (SSE) | ✅ Supported |
| Multi-model serving | ✅ Supported |
| Integrated KV-aware routing | ✅ Supported |
| Tool calling | ✅ Supported |
| TLS (HTTPS) | ✅ Supported |
| Swagger UI (`/docs`) | ✅ Supported |
| NVIDIA request extensions (`nvext`) | ✅ Supported |

## Quick Start

### Prerequisites

- Dynamo platform installed
- `etcd` and `nats-server -js` running
- At least one backend worker registered

### HTTP Frontend

```bash
python -m dynamo.frontend --http-port 8000
```

This starts an OpenAI-compatible HTTP server with integrated pre/post processing and routing. Backends are auto-discovered when they call `register_model`.

The frontend does the pre and post processing. To do this it will need access to the model configuration files: `config.json`, `tokenizer.json`, `tokenizer_config.json`, etc. It does not need the weights.

Frontend will download the files it needs from Hugging Face, no setup is required. However we recommend setting up [modelexpress-server](https://github.com/ai-dynamo/modelexpress) and a shared folder such as a Kubernetes PVC. This ensures the model is only downloaded once across the whole cluster.

If the model is not available on Hugging Face, such as a private or customized model, you will need to make the model files available locally at the same file path as on the backend. The backend's `--model-path <here>` will need to exist on the frontend and contain at least the configuration (JSON) files.

### KServe gRPC Frontend

```bash
python -m dynamo.frontend --kserve-grpc-server
```

See the [Frontend Guide](/dynamo/v1.3.0/components/frontend/frontend-guide) for KServe-specific configuration and message formats.

### Kubernetes

```yaml
apiVersion: nvidia.com/v1alpha1
kind: DynamoGraphDeployment
metadata:
  name: frontend-example
spec:
  graphs:
    - name: frontend
      replicas: 1
      services:
        - name: Frontend
          image: nvcr.io/nvidia/dynamo/dynamo-vllm:latest
          command:
            - python
            - -m
            - dynamo.frontend
            - --http-port
            - "8000"
```

## Configuration

| Parameter | Default | Description |
|-----------|---------|-------------|
| `--http-port` | 8000 | HTTP server port |
| `--kserve-grpc-server` | false | Enable KServe gRPC server |
| `--router-mode` | `round-robin` | Routing strategy: `round-robin`, `random`, `power-of-two`, `kv`, `direct`, `least-loaded`, `device-aware-weighted` (`power-of-two` and `least-loaded` use synchronous prefill fallback in disaggregated prefill mode) |

See the [Frontend Guide](/dynamo/v1.3.0/components/frontend/frontend-guide) for full configuration options.

## Next Steps

| Document | Description |
|----------|-------------|
| [Configuration Reference](/dynamo/v1.3.0/components/frontend/configuration-reference) | All CLI arguments, env vars, and HTTP endpoints |
| [Frontend Guide](/dynamo/v1.3.0/components/frontend/frontend-guide) | KServe gRPC configuration and integration |
| [NVIDIA Request Extensions (nvext)](/dynamo/v1.3.0/additional-resources/nvidia-request-extensions-nvext) | Custom request fields for routing hints and cache control |
| [Python Route Extensions](/dynamo/v1.3.0/components/frontend/python-route-extensions) | Register additional HTTP routes on the frontend from a Python package |
| [Router Documentation](/dynamo/v1.3.0/components/router) | KV-aware routing configuration |

> API gateway that exposes OpenAI HTTP and KServe gRPC endpoints and handles request preprocessing, routing, and response formatting.

## Docs

- [Frontend Guide](https://docs.nvidia.com/dynamo/v1.3.0/components/frontend/frontend-guide.md): Configures the KServe gRPC frontend, including supported endpoints, gRPC tuning, and registering OpenAI or tensor-based backends.
- [Frontend Configuration Reference](https://docs.nvidia.com/dynamo/v1.3.0/components/frontend/configuration-reference.md): Complete reference for all frontend CLI arguments, environment variables, and HTTP endpoints
- [Python Route Extensions](https://docs.nvidia.com/dynamo/v1.3.0/components/frontend/python-route-extensions.md)
- [Tokenizer](https://docs.nvidia.com/dynamo/v1.3.0/components/frontend/tokenizer.md): Selects between the default HuggingFace and fastokens tokenizer backends for BPE models served through the Dynamo Frontend.