Run Guardrailed InferenceGuardrails API Server

Run the Guardrails Server

View as Markdown

The NeMo Guardrails server loads configurations from a local directory at startup and exposes an HTTP API to use them. The server uses FastAPI and includes a built-in Chat UI for testing.

Start the Server

Launch the server using the nemoguardrails CLI:

nemoguardrails server --config examples/configs

For more information about the available options, see CLI: Server.

The server supports two modes depending on your folder structure: multi-config mode and single-config mode.

Multi-Config Mode

When the --config path points to a folder containing multiple sub-folders, each sub-folder with a config.yml file becomes an available configuration. The sub-folder name becomes the config_id.

examples/configs/          # --config points here
├── content_safety/        # config_id: "content_safety"
│   ├── rails.co
│   └── config.yml
├── jailbreak_detection/   # config_id: "jailbreak_detection"
│   ├── flows.co
│   └── config.yml
└── topic_safety/          # config_id: "topic_safety"
    └── config.yml

  1. Start the server in multi-config mode:

    nemoguardrails server --config examples/configs

  2. List available configurations.

    curl http://localhost:8000/v1/rails/configs

    The endpoint returns the list of available configurations.

    [
      {"id": "content_safety"},
      {"id": "jailbreak_detection"},
      {"id": "topic_safety"}
    ]

Single-Config Mode

When the --config path points directly to a folder containing a config.yml file, the server runs in single-config mode. The folder name becomes the only available config_id.

examples/configs/content_safety/   # --config points here
├── rails.co
└── config.yml                     # config_id: "content_safety"

  1. Start the server in single-config mode:

    nemoguardrails server --config examples/configs/content_safety

  2. List available configurations.

    curl http://localhost:8000/v1/rails/configs

    The endpoint returns the single configuration named content_safety.

Examples

The following examples show how to start the server with different options.

Start with Default Settings

The following command starts the server with default settings.

nemoguardrails server

The server starts on port 8000 and looks for a ./config folder in the current directory. If not found, it uses the built-in example configurations.

Start with Custom Port

You can use the --port flag to start the server on a custom port.

nemoguardrails server --config examples/configs --port 8080

Start with a Default Configuration

Use the following command to start the server with a default configuration within a multi-config folder. For example, when you use the provided example configurations (examples/configs), you can set the default configuration to content_safety as follows.

nemoguardrails server --config examples/configs --default-config-id content_safety

Chat completions requests without a config_id use the content_safety configuration by default.

Start in Development Mode

You can add the --auto-reload flag to the server to automatically reload when configuration files change.

nemoguardrails server --config ./configs --auto-reload

Use --auto-reload only in development environments. It is not recommended for production.

Model Provider Configuration

When the model field is specified in a chat completion request, that model name replaces the name of the main model in your configuration. The rest of the configured main model still applies, including engine, api_key_env_var, mode, and parameters. Use the following environment variables to override the provider and endpoint the server uses for that model.

:header-rows: 1
:widths: 35 65
* - Environment Variable
  - Description
* - `MAIN_MODEL_ENGINE`
  - The LLM engine to use (e.g., `"openai"`, `"nim"`, `"vllm"`, `"anthropic"`). Overrides the `engine` of the configured main model. Defaults to that configured `engine`, or to `"openai"` if your configuration declares no main model.
* - `MAIN_MODEL_BASE_URL`
  - Base URL for the LLM provider. Overrides `parameters.base_url` of the configured main model. Use this for self-hosted models (e.g., `"http://localhost:8080/v1"`).

Set the appropriate API key for your provider:

# For NVIDIA-hosted models
export MAIN_MODEL_ENGINE="nim"
export MAIN_MODEL_BASE_URL="https://integrate.api.nvidia.com"
export NVIDIA_API_KEY="your-nvidia-api-key"
# For OpenAI models
export MAIN_MODEL_ENGINE="openai"
export MAIN_MODEL_BASE_URL="https://api.openai.com/v1"
export OPENAI_API_KEY="your-openai-api-key"
# For Anthropic models
export MAIN_MODEL_ENGINE="anthropic"
export MAIN_MODEL_BASE_URL="https://api.anthropic.com"
export ANTHROPIC_API_KEY="your-anthropic-api-key"
# For Azure OpenAI (also accepts "azure_openai" as engine name)
export MAIN_MODEL_ENGINE="azure"
export AZURE_OPENAI_API_KEY="your-azure-api-key"
export AZURE_OPENAI_ENDPOINT="https://your-resource.openai.azure.com"
export AZURE_OPENAI_API_VERSION="2024-06-01"  # optional, defaults to 2024-06-01
# For Cohere models
export MAIN_MODEL_ENGINE="cohere"
export COHERE_API_KEY="your-cohere-api-key"
# export COHERE_BASE_URL="https://custom-endpoint.example.com"  # optional
# For self-hosted models (e.g., vLLM, NIM, TRT-LLM)
export MAIN_MODEL_ENGINE="vllm"
export MAIN_MODEL_BASE_URL="http://localhost:8080/v1"

CORS Configuration

To enable your guardrails server to receive requests from browser-based applications, configure CORS using environment variables:

:header-rows: 1
:widths: 40 60
* - Environment Variable
  - Description
* - `NEMO_GUARDRAILS_SERVER_ENABLE_CORS`
  - Set to `true` to enable CORS. Default: `false`.
* - `NEMO_GUARDRAILS_SERVER_ALLOWED_ORIGINS`
  - Comma-separated list of allowed origins. Default: `*`.

Example:

export NEMO_GUARDRAILS_SERVER_ENABLE_CORS=true
export NEMO_GUARDRAILS_SERVER_ALLOWED_ORIGINS=http://localhost:3000,https://myapp.com
nemoguardrails server --config ./configs

Chat UI

The server includes a built-in Chat UI for testing guardrails configurations. Access it at http://localhost:8000/ after starting the server.

The Chat UI is for internal testing only. For production deployments, disable it using the --disable-chat-ui flag.

Health Check

The server exposes a liveness endpoint at GET /v1/health (also available as GET /healthz) for orchestrators, load balancers, and container runtimes. It returns HTTP 200 with the application/health+json media type and a body of {"status": "pass"} whenever the server process is running and able to serve requests.

curl http://localhost:8000/v1/health
{"status": "pass"}

The check is intentionally shallow: it reflects only that the server process is up, and does not verify guardrails configurations, the model provider, or any datastore. Use this endpoint rather than GET / for health probes, because GET / redirects to the chat UI when the UI is enabled. When the server runs behind a path prefix (--prefix), the endpoints are served at <prefix>/v1/health and <prefix>/healthz.