List Available Models

Use the GET /v1/models endpoint to retrieve the list of LLM models available from the configured provider. This endpoint is compatible with the OpenAI Models API and proxies the request to the upstream model provider configured via environment variables.

Request

$
curl http://localhost:8000/v1/models

No request body or query parameters are required.

Response

The endpoint returns a standard OpenAI models list:

1
{
2
  "data": [
3
    {
4
      "id": "meta/llama-3.1-8b-instruct",
5
      "object": "model",
6
      "created": 1700000000,
7
      "owned_by": "system"
8
    },
9
    {
10
      "id": "meta/llama-3.1-70b-instruct",
11
      "object": "model",
12
      "created": 1700000000,
13
      "owned_by": "system"
14
    }
15
  ]
16
}

Using Python

1
import requests
2
3
base_url = "http://localhost:8000"
4
5
response = requests.get(f"{base_url}/v1/models")
6
models = response.json()
7
8
print("Available models:")
9
for model in models["data"]:
10
    print(f"  - {model['id']}")

Using the OpenAI Python SDK

1
from openai import OpenAI
2
3
client = OpenAI(
4
    base_url="http://localhost:8000/v1",
5
    api_key="not-used"
6
)
7
8
models = client.models.list()
9
10
for model in models.data:
11
    print(model.id)

Provider Configuration

The server determines which upstream provider to query based on the MAIN_MODEL_ENGINE and related environment variables. The following providers are supported.

OpenAI

$
export MAIN_MODEL_ENGINE="openai"
$
export MAIN_MODEL_BASE_URL="https://api.openai.com/v1"
$
export OPENAI_API_KEY="your-openai-api-key"

NVIDIA NIM

$
export MAIN_MODEL_ENGINE="nim"
$
export MAIN_MODEL_BASE_URL="http://localhost:8080/v1"

vLLM / TRT-LLM

$
export MAIN_MODEL_ENGINE="vllm"  # or "trt_llm"
$
export MAIN_MODEL_BASE_URL="http://localhost:8080/v1"

Anthropic

$
export MAIN_MODEL_ENGINE="anthropic"
$
export ANTHROPIC_API_KEY="your-anthropic-api-key"

Azure OpenAI

The engine name azure_openai is also accepted as an alias for azure.

$
export MAIN_MODEL_ENGINE="azure"
$
export AZURE_OPENAI_API_KEY="your-azure-api-key"
$
export AZURE_OPENAI_ENDPOINT="https://your-resource.openai.azure.com"
$
export AZURE_OPENAI_API_VERSION="2024-06-01"  # optional, defaults to 2024-06-01

Cohere

$
export MAIN_MODEL_ENGINE="cohere"
$
export COHERE_API_KEY="your-cohere-api-key"
$
# Optional: override the Cohere API base URL (defaults to https://api.cohere.com)
$
# export COHERE_BASE_URL="https://custom-cohere-endpoint.example.com"

Custom OpenAI-Compatible Endpoint

For any provider that exposes an OpenAI-compatible /v1/models endpoint, set MAIN_MODEL_BASE_URL:

$
export MAIN_MODEL_ENGINE="my-custom-engine"
$
export MAIN_MODEL_BASE_URL="http://my-provider:8080/v1"

The server falls back to querying MAIN_MODEL_BASE_URL when the engine is not in the built-in provider table.

Authentication

The endpoint forwards the Authorization header from the incoming request to the upstream provider. If no Authorization header is present, the server uses the API key from the appropriate environment variable for the configured engine.

Error Responses

Related Topics

• Run the Guardrails Server
• Chat with a Guardrailed Model
• List Models