Run Guardrailed InferenceGuardrails API Server

Overview of the NVIDIA NeMo Guardrails Library API Server

View as Markdown

The NVIDIA NeMo Guardrails library API server provides the following capabilities:

  • Loads guardrails configurations at startup.
  • Exposes an OpenAI-compatible REST API for chat completions and model listing.
  • Works with the OpenAI Python SDK. Use OpenAI(base_url="http://localhost:8000/v1").
  • Includes a built-in chat UI for testing.
  • Supports multiple configurations and combines them for each request.

Quick Start

The following steps show how to start the NVIDIA NeMo Guardrails library API server with the provided configuration files and send test requests to the endpoints.

Prerequisites

Meet the following prerequisites before you use the NVIDIA NeMo Guardrails library API server.

  1. Install the NVIDIA NeMo Guardrails library with the server extra. For instructions, refer to Extra Dependencies.

    $pip install nemoguardrails[server]

  2. Set the environment variable for your NVIDIA API key.

    1export NVIDIA_API_KEY="your-nvidia-api-key"

    This key is required to access NVIDIA-hosted models on build.nvidia.com. The provided example configurations and code examples throughout the documentation use NVIDIA-hosted models.

Start the Server

Follow these steps to start the server:

  1. Point the server to a parent directory that contains multiple configuration subdirectories:

    1$ cd Guardrails
    2$ nemoguardrails server --config examples/configs

  2. To check if the server is running and list the available configurations, use the following command:

    1$ curl http://localhost:8000/v1/rails/configs
    2
    3[
    4  {"id": "content_safety"},
    5  {"id": "jailbreak_detection"},
    6  {"id": "topic_safety"},
    7  {"id": "llama_guard"},
    8  ...
    9]

Each subdirectory that contains a config.yml or config.yaml file becomes an available configuration ID.

Send a Request

Send a chat completion request to the server:

$curl -X POST http://localhost:8000/v1/chat/completions \
>  -H "Content-Type: application/json" \
>  -d '{
>    "model": "meta/llama-3.1-8b-instruct",
>    "messages": [{"role": "user", "content": "Hello!"}],
>    "guardrails": {
>      "config_id": "content_safety"
>    }
>  }'

View the Chat UI

Open http://localhost:8000 in your browser to access the built-in chat UI for testing.