About Guardrails#

Learn how to add checks and content moderation to a model by making requests to the NVIDIA NeMo Guardrails microservice.

How NeMo Guardrails Interacts with LLMs#

NVIDIA NeMo Guardrails sits between your application and the LLM:

Guardrails between applications and LLMs

A single NeMo Guardrails microservice instance can serve multiple applications and work with multiple guardrail configurations and LLMs. To use the NeMo Guardrails microservice, you must configure your application to use the NeMo Guardrails microservice endpoint, rather than the original LLM endpoint.

Guardrails Workflow#

Before you get started:

  • Know the fine-tuned or foundation LLM model that you want to add guardrails to.

  • Know the roles, such as user, assistant, and system that the model uses.

  • Know the request options, such as max_tokens, stream, and temperature for the model.

Develop the guardrails:

  1. Create a guardrails configuration that defines the safety checks for input and output.

  2. Check the guardrails.

  3. Run inference with the guardrails.

Task Guides#

The following guides provide detailed information on how to perform common guardrail operations.

Manage Guardrail Configurations

Manage guardrail configurations.

Manage Guardrail Configs
Check Guardrails

Check content with input and output guardrails.

Checking a Guardrail
Run Inference with Guardrails

Run inference and apply safety checks.

Running Inference with Guardrails

Tutorials#

Follow these tutorials to learn how to accomplish common guardrail tasks.

Integrate with NemoGuard NIM

Use NemoGuard NIM for content safety and topic control

nemo-guardrails

Integrating with NemoGuard NIM Microservices
Develop Guardrails in Docker

Run the NeMo Guardrails container locally

nemo-guardrails

Deploying with Docker
Custom Dependencies

Build an image with custom dependencies

nemo-guardrails

Custom Dependencies
Custom HTTP Headers

Add custom HTTP headers to requests

nemo-guardrails

Custom HTTP Headers
Multiple NIM for LLMs

Use multiple NIM microservices for LLMs

nemo-guardrails nim-for-llms

Working with Multiple NIM for LLMs
Custom LLM Providers

Send requests to custom LLM providers

nemo-guardrails

Custom LLM Providers

Limitations and Restrictions#

The following list identifies the key differences between the NeMo Guardrails microservice and the NeMo Guardrails open-source toolkit.

  • Certain toolkit server features are unavailable in the microservice, such as threads and the chat user interface.

  • The microservice supports Colang 1.0 guardrail configurations only.

  • The microservice supports NVIDIA NIM for LLMs and LLMs with an OpenAI compatible API as LLM providers.

  • The API is OpenAI compatible, with the addition of the guardrails field in the request body and the guardrails_data field in the response body.

  • In the microservice, the LLM provider is determined by the model field that is specified in the HTTP request. In the toolkit, the LLM provider is determined by the guardrail configuration.

  • The microservice supports a default guardrail configuration that is used when no guardrail configuration is specified as part of an HTTP request.