Run the Guardrails Server

The NeMo Guardrails server loads configurations from a local directory at startup and exposes an HTTP API to use them. The server uses FastAPI and includes a built-in Chat UI for testing.

Start the Server

Launch the server using the nemoguardrails CLI:

$
nemoguardrails server --config examples/configs

For more information about the available options, see CLI: Server.

Link Guardrail Configurations to the Server

The server supports two modes depending on your folder structure: multi-config mode and single-config mode.

Multi-Config Mode

When the --config path points to a folder containing multiple sub-folders, each sub-folder with a config.yml file becomes an available configuration. The sub-folder name becomes the config_id.

examples/configs/          # --config points here
├── content_safety/        # config_id: "content_safety"
│   ├── rails.co
│   └── config.yml
├── jailbreak_detection/   # config_id: "jailbreak_detection"
│   ├── flows.co
│   └── config.yml
└── topic_safety/          # config_id: "topic_safety"
    └── config.yml

1. Start the server in multi-config mode:

   $
   nemoguardrails server --config examples/configs

2. List available configurations.

   $
   curl http://localhost:8000/v1/rails/configs

   The endpoint returns the list of available configurations.

   1
   [
   2
     {"id": "content_safety"},
   3
     {"id": "jailbreak_detection"},
   4
     {"id": "topic_safety"}
   5
   ]

Single-Config Mode

When the --config path points directly to a folder containing a config.yml file, the server runs in single-config mode. The folder name becomes the only available config_id.

examples/configs/content_safety/   # --config points here
├── rails.co
└── config.yml                     # config_id: "content_safety"

1. Start the server in single-config mode:

   $
   nemoguardrails server --config examples/configs/content_safety

2. List available configurations.

   $
   curl http://localhost:8000/v1/rails/configs

   The endpoint returns the single configuration named content_safety.

Examples

The following examples show how to start the server with different options.

Start with Default Settings

The following command starts the server with default settings.

$
nemoguardrails server

The server starts on port 8000 and looks for a ./config folder in the current directory. If not found, it uses the built-in example configurations.

Start with Custom Port

You can use the --port flag to start the server on a custom port.

$
nemoguardrails server --config examples/configs --port 8080

Start with a Default Configuration

Use the following command to start the server with a default configuration within a multi-config folder. For example, when you use the provided example configurations (examples/configs), you can set the default configuration to content_safety as follows.

$
nemoguardrails server --config examples/configs --default-config-id content_safety

Chat completions requests without a config_id use the content_safety configuration by default.

Start in Development Mode

You can add the --auto-reload flag to the server to automatically reload when configuration files change.

$
nemoguardrails server --config ./configs --auto-reload

Model Provider Configuration

When the model field is specified in a chat completion request, the server uses environment variables to determine which LLM provider and endpoint to use.

1
:header-rows: 1
2
:widths: 35 65
3
4
* - Environment Variable
5
  - Description
6
7
* - `MAIN_MODEL_ENGINE`
8
  - The LLM engine to use (e.g., `"openai"`, `"nim"`, `"vllm"`, `"anthropic"`). Default: `"openai"`.
9
10
* - `MAIN_MODEL_BASE_URL`
11
  - Base URL for the LLM provider. Use this for self-hosted models (e.g., `"http://localhost:8080/v1"`).

Set the appropriate API key for your provider:

$
# For NVIDIA-hosted models
$
export MAIN_MODEL_ENGINE="nim"
$
export MAIN_MODEL_BASE_URL="https://integrate.api.nvidia.com"
$
export NVIDIA_API_KEY="your-nvidia-api-key"
$
$
# For OpenAI models
$
export MAIN_MODEL_ENGINE="openai"
$
export MAIN_MODEL_BASE_URL="https://api.openai.com/v1"
$
export OPENAI_API_KEY="your-openai-api-key"
$
$
# For Anthropic models
$
export MAIN_MODEL_ENGINE="anthropic"
$
export MAIN_MODEL_BASE_URL="https://api.anthropic.com"
$
export ANTHROPIC_API_KEY="your-anthropic-api-key"
$
$
# For Azure OpenAI (also accepts "azure_openai" as engine name)
$
export MAIN_MODEL_ENGINE="azure"
$
export AZURE_OPENAI_API_KEY="your-azure-api-key"
$
export AZURE_OPENAI_ENDPOINT="https://your-resource.openai.azure.com"
$
export AZURE_OPENAI_API_VERSION="2024-06-01"  # optional, defaults to 2024-06-01
$
$
# For Cohere models
$
export MAIN_MODEL_ENGINE="cohere"
$
export COHERE_API_KEY="your-cohere-api-key"
$
# export COHERE_BASE_URL="https://custom-endpoint.example.com"  # optional
$
$
# For self-hosted models (e.g., vLLM, NIM, TRT-LLM)
$
export MAIN_MODEL_ENGINE="vllm"
$
export MAIN_MODEL_BASE_URL="http://localhost:8080/v1"

CORS Configuration

To enable your guardrails server to receive requests from browser-based applications, configure CORS using environment variables:

1
:header-rows: 1
2
:widths: 40 60
3
4
* - Environment Variable
5
  - Description
6
7
* - `NEMO_GUARDRAILS_SERVER_ENABLE_CORS`
8
  - Set to `true` to enable CORS. Default: `false`.
9
10
* - `NEMO_GUARDRAILS_SERVER_ALLOWED_ORIGINS`
11
  - Comma-separated list of allowed origins. Default: `*`.

Example:

$
export NEMO_GUARDRAILS_SERVER_ENABLE_CORS=true
$
export NEMO_GUARDRAILS_SERVER_ALLOWED_ORIGINS=http://localhost:3000,https://myapp.com
$
nemoguardrails server --config ./configs

Chat UI

The server includes a built-in Chat UI for testing guardrails configurations. Access it at http://localhost:8000/ after starting the server.

Related Topics

• Chat with a Guardrailed Model
• List Guardrail Configurations
• Create Chat Completion