Skip to navigation

Configuration YAML Schema Reference

View as MarkdownOpen in Claude

This reference documents all configuration options for config.yml, derived from the authoritative Pydantic schema in nemoguardrails/rails/llm/config.py.


Models Configuration

The models key defines LLM providers and models used by the NVIDIA NeMo Guardrails library.

Model Schema

models:
- type: main # Required: Model type
engine: openai # Required: LLM provider
model: gpt-4o # Required: Model name
mode: chat # Optional: "chat" or "text" (default: "chat")
api_key_env_var: OPENAI_KEY # Optional: Environment variable for API key
parameters: # Optional: Provider-specific parameters
temperature: 0.7
max_tokens: 1000
cache: # Optional: Caching configuration
enabled: false
maxsize: 50000

Model Attributes

AttributeTypeRequiredDescription
models.api_key_env_varstringEnvironment variable containing API key
models.cacheobjectCache configuration for this model
models.enginestring✓LLM provider (see Engines)
models.modestringCompletion mode: chat or text (default: chat)
models.modelstring✓Model name (can also be in parameters.model_name)
models.parametersobjectProvider-specific parameters. For engines served by the built-in client, such as any OpenAI-compatible endpoint, the runtime forwards parameters to the OpenAI-compatible HTTP request. Examples include temperature, max_tokens, base_url, api_key, default_query, and default_headers. default_headers and default_query configure the HTTP request rather than the request body. Refer to Custom HTTP Headers and Model-Level Query Parameters. For engines served by LangChain, opt in with NEMOGUARDRAILS_LLM_FRAMEWORK=langchain; the runtime forwards parameters to the underlying LangChain class. For the engine-by-engine matrix, refer to Inference Providers.
models.typestring✓Model identifier (see Model Types)

Model Types

The type field is a free-form string identifier. Certain types have special handling in the runtime, while custom types can be defined and referenced in flows via $model=<type>.

Reserved Types

These types have special handling in the runtime:

TypeDescription
embeddingsEmbedding model for knowledge base and similarity search
jailbreak_detectionJailbreak detection model (used with NIM)
mainPrimary application LLM for conversation

Commonly-Used Types

The following types are commonly used with guardrails:

TypeDescriptionUsage Example in Flows
content_safetyContent safety modelcontent safety check input $model=content_safety
llama_guardLlama Guard content moderationllama guard check input $model=llama_guard
topic_controlTopic control modeltopic safety check input $model=topic_control

Custom Types

You can define any custom type and reference it in flows. For example:

models:
- type: my_safety_model
engine: self-hosted
model: my-org/custom-safety-model
rails:
input:
flows:
- content safety check input $model=my_safety_model

The runtime validates that any $model=<type> reference in flows has a matching model defined in the configuration.

Engines

Starting with v0.22, the library serves engines through either the built-in OpenAI-compatible client or LangChain. Use the built-in client whenever the underlying wire protocol is OpenAI-compatible. Opt into LangChain only for engines whose API is not OpenAI-compatible, such as Vertex AI, Anthropic, Cohere, and the in-process Hugging Face pipeline. For the full mapping, refer to Inference Providers. For migration recipes, refer to Migrating to 0.22.

Built-in Engines

These engines work with pip install nemoguardrails and do not require extra provider packages. Pass parameters.base_url to point at a self-hosted or alternative endpoint.

EngineDescription
azure, azure_openaiAzure OpenAI models with key-based authentication (azure_endpoint or base_url, azure_deployment, and api_version)
nimNVIDIA NIM microservices
nvidia_ai_endpointsAlias for nim
ollamaOllama OpenAI-compatible endpoint at http://localhost:11434/v1
openaiOpenAI public API or any OpenAI-compatible endpoint using parameters.base_url

For OpenAI-compatible providers without a dedicated engine entry (vLLM, TGI, OpenRouter, Together.ai, Fireworks.ai, Groq, DeepSeek, llama.cpp server, and similar), use engine: openai with parameters.base_url and parameters.api_key.

LangChain Engines

To use one of these engines, set NEMOGUARDRAILS_LLM_FRAMEWORK=langchain and install the matching langchain-* provider package.

EngineDescription
anthropicAnthropic Claude models
cohereCohere models
google_genaiGoogle Generative AI through LangChain (requires langchain-google-genai)
huggingface_endpointHugging Face Inference Endpoints (default text-generation schema; if your endpoint exposes /v1/chat/completions, prefer engine: openai with parameters.base_url instead)
huggingface_hubHugging Face Hub models
huggingface_pipelineIn-process Hugging Face pipeline
self_hostedGeneric self-hosted LangChain wrapper
trt_llmTensorRT-LLM in-process
vertexaiGoogle Vertex AI through LangChain (requires langchain-google-vertexai)
vllm_openaiLegacy LangChain wrapper for vLLM. For new configurations, prefer engine: openai with parameters.base_url

Embedding Engines

EngineDescription
FastEmbedFastEmbed (default)
nimNVIDIA NIM embeddings
openaiOpenAI embeddings

Model Cache Configuration

models:
- type: content_safety
engine: nim
model: nvidia/llama-3.1-nemotron-safety-guard-8b-v3
cache:
enabled: true
maxsize: 50000
stats:
enabled: false
log_interval: null
AttributeTypeDefaultDescription
models.cache.enabledbooleanfalseEnable caching for this model
models.cache.maxsizeinteger50000Maximum cache entries
models.cache.stats.enabledbooleanfalseEnable cache statistics tracking
models.cache.stats.log_intervalfloatnullSeconds between stats logging

Rails Configuration

The rails key configures guardrails that control LLM behavior.

Rails Schema

rails:
input:
parallel: false
flows:
- self check input
- check jailbreak
output:
parallel: false
flows:
- self check output
streaming:
enabled: false
chunk_size: 200
context_size: 50
stream_first: true
retrieval:
flows:
- check retrieval sensitive data
dialog:
single_call:
enabled: false
fallback_to_multiple_calls: true
user_messages:
embeddings_only: false
actions:
instant_actions: []
tool_call:
flows: []
parallel: false
tool_result:
flows: []
parallel: false
config:
# Rail-specific configurations

Rail Types

The following table summarizes the available rail types and their trigger points.

Rail TypeTrigger PointPurpose
Dialog railsAfter canonical form is computedControl conversation flow
Execution railsBefore/after action executionControl tool and action calls
Input railsWhen user input is receivedValidate, filter, or modify user input
Output railsWhen LLM generates outputValidate, filter, or modify bot responses
Retrieval railsAfter RAG retrieval completesProcess retrieved chunks

The following diagram shows the guardrails process described in the table above in detail.

Diagram showing the programmable guardrails flow

Input Rails

Process user messages before they reach the LLM.

rails:
input:
parallel: false # Execute flows in parallel
flows:
- self check input
- check jailbreak
- mask sensitive data on input
AttributeTypeDefaultDescription
rails.input.flowslist[]Names of flows that implement input rails
rails.input.parallelbooleanfalseExecute input rails in parallel

Built-in Input Flows

FlowDescription
content safety check inputNVIDIA content safety model
detect sensitive data on inputDetect and block PII
jailbreak detection heuristicsJailbreak detection heuristics
jailbreak detection modelNIM-based jailbreak detection
llama guard check inputLlamaGuard content moderation
mask sensitive data on inputMask PII in user input
self check inputLLM-based policy compliance check
topic safety check inputTopic control model

Output Rails

Process LLM responses before returning to users.

rails:
output:
parallel: false
flows:
- self check output
- self check facts
streaming:
enabled: false
chunk_size: 200
context_size: 50
stream_first: true
AttributeTypeDefaultDescription
rails.output.flowslist[]Names of flows that implement output rails
rails.output.parallelbooleanfalseExecute output rails in parallel
rails.output.streamingobjectStreaming output configuration

Output Streaming Configuration

AttributeTypeDefaultDescription
rails.output.streaming.chunk_sizeinteger200Tokens per processing chunk
rails.output.streaming.context_sizeinteger50Tokens carried from previous chunk
rails.output.streaming.enabledbooleanfalseEnable streaming mode
rails.output.streaming.stream_firstbooleantrueStream before applying output rails

Built-in Output Flows

FlowDescription
content safety check outputNVIDIA content safety model
injection detectionInjection detection (SQL, XSS, code, template)
llama guard check outputLlamaGuard content moderation
mask sensitive data on outputMask PII in output
self check factsFact verification
self check hallucinationHallucination detection
self check outputLLM-based policy compliance check

Retrieval Rails

Process chunks retrieved from knowledge base.

rails:
retrieval:
flows:
- check retrieval sensitive data

Dialog Rails

Control conversation flow after user intent is determined.

rails:
dialog:
single_call:
enabled: false
fallback_to_multiple_calls: true
user_messages:
embeddings_only: false
embeddings_only_similarity_threshold: null
embeddings_only_fallback_intent: null
AttributeTypeDefaultDescription
rails.dialog.single_call.enabledbooleanfalseUse single LLM call for intent + response
rails.dialog.single_call.fallback_to_multiple_callsbooleantrueFall back if single call fails
rails.dialog.user_messages.embeddings_onlybooleanfalseUse only embeddings for intent matching

Execution Rails

Control tool and action invocations.

Action Rails

Control custom action and tool invocations.

rails:
actions:
instant_actions:
- action_name_1
- action_name_2

Tool Rails

The tool_call rails check the tool calls a model emits, and the tool_result rails check the tool results returned to the model. LLMRails runs the Colang flows configured in each section. IORails provides the following built-in structural validation flows:

rails:
tool_call:
flows:
- tool call validation
tool_result:
flows:
- tool result validation

With IORails, each section’s flows field accepts only its own built-in flow name: tool_call accepts tool call validation, and tool_result accepts tool result validation. The tool_output and tool_input fields are deprecated aliases for tool_call and tool_result, respectively. Since each section names at most one flow, parallel has no observable effect on them: they always run one at a time regardless of the setting.

Each section also accepts a per_tool field, keyed by tool name, scoping a check to one specific tool’s calls or results, for example:

rails:
config:
regex_detection:
tool_call:
run_sql:
patterns:
- "DROP\\s+TABLE"
tool_call:
per_tool:
run_sql:
- regex check tool call

Unlike the fixed flows above, a request can run per-tool checks for several tool calls, or several flows for one tool, at once. parallel is honored here: set it to run those checks concurrently instead of one after another. When multiple checks run concurrently and more than one would block, only the first to complete and block is reported; the same conversation history can therefore produce a different blocking rail or reason across runs, depending on which check’s judge model or backend responds fastest. For configuration details and behavior, see Tool Calling.

Rails Config Section

The rails.config section contains configuration for specific built-in rails.

Jailbreak Detection

rails:
config:
jailbreak_detection:
# Heuristics-based detection
server_endpoint: null
length_per_perplexity_threshold: 89.79
prefix_suffix_perplexity_threshold: 1845.65
# NIM-based detection
nim_base_url: "http://localhost:8000/v1/"
nim_server_endpoint: "classify"
api_key_env_var: "JAILBREAK_KEY"
AttributeTypeDefaultDescription
rails.config.jailbreak_detection.api_keystringnullAPI key (not recommended)
rails.config.jailbreak_detection.api_key_env_varstringnullEnvironment variable for API key
rails.config.jailbreak_detection.length_per_perplexity_thresholdfloat89.79Length/perplexity threshold
rails.config.jailbreak_detection.nim_base_urlstringnullNIM base URL (e.g., http://localhost:8000/v1)
rails.config.jailbreak_detection.nim_server_endpointstring"classify"NIM endpoint path
rails.config.jailbreak_detection.prefix_suffix_perplexity_thresholdfloat1845.65Prefix/suffix perplexity threshold
rails.config.jailbreak_detection.server_endpointstringnullHeuristics model endpoint

Sensitive Data Detection (Presidio)

rails:
config:
sensitive_data_detection:
recognizers: []
input:
entities:
- PERSON
- EMAIL_ADDRESS
- PHONE_NUMBER
- CREDIT_CARD
mask_token: "*"
score_threshold: 0.2
output:
entities:
- PERSON
- EMAIL_ADDRESS
retrieval:
entities: []
AttributeTypeDefaultDescription
rails.config.sensitive_data_detection.input/output/retrieval.entitieslist[]Entity types to detect
rails.config.sensitive_data_detection.input/output/retrieval.mask_tokenstring"*"Token for masking
rails.config.sensitive_data_detection.input/output/retrieval.score_thresholdfloat0.2Detection confidence threshold
rails.config.sensitive_data_detection.recognizerslist[]Custom Presidio recognizers

Injection Detection

rails:
config:
injection_detection:
injections:
- sqli
- template
- code
- xss
action: reject # "reject" or "omit"
yara_path: ""
yara_rules: {}
AttributeTypeDefaultDescription
rails.config.injection_detection.actionstring"reject"Action: reject or omit
rails.config.injection_detection.injectionslist[]Injection types: sqli, template, code, xss
rails.config.injection_detection.yara_pathstring""Custom YARA rules path
rails.config.injection_detection.yara_rulesobject{}Inline YARA rules

Fact Checking

rails:
config:
fact_checking:
parameters:
endpoint: "http://localhost:5000"
fallback_to_self_check: false

Content Safety

rails:
config:
content_safety:
multilingual:
enabled: false
refusal_messages:
en: "Sorry, I cannot help with that."
es: "Lo siento, no puedo ayudar con eso."

The multilingual feature supports the following languages:

LanguageCode
Arabicar
Chinesezh
Englishen
Frenchfr
Germande
Hindihi
Japaneseja
Spanishes
Thaith

If the detected language is not in this list, English is used as the fallback. For more information, refer to Multilingual Content Safety.

Third-Party Integrations

AutoAlign
rails:
config:
autoalign:
parameters: {}
input:
guardrails_config: {}
output:
guardrails_config: {}

For more information, refer to AutoAlign Integration.

Patronus
rails:
config:
patronus:
input:
evaluate_config:
success_strategy: all_pass # or any_pass
params: {}
output:
evaluate_config:
success_strategy: all_pass
params: {}

For more information, refer to Patronus Evaluate API Integration.

Clavata
rails:
config:
clavata:
server_endpoint: "https://gateway.app.clavata.ai:8443"
policies: {}
label_match_logic: ANY # or ALL
input:
policy: "policy_alias"
labels: []
output:
policy: "policy_alias"
labels: []

For more information, refer to Clavata Integration.

Pangea AI Guard
rails:
config:
pangea:
input:
recipe: "recipe_key"
output:
recipe: "recipe_key"

For more information, refer to Pangea AI Guard Integration.

Trend Micro
rails:
config:
trend_micro:
v1_url: "https://api.xdr.trendmicro.com/beta/aiSecurity/guard"
api_key_env_var: "TREND_MICRO_API_KEY"

For more information, refer to Trend Micro Integration.

Cisco AI Defense
rails:
config:
ai_defense:
timeout: 30.0
fail_open: false

For more information, refer to Cisco AI Defense Integration.

Private AI
rails:
config:
private_ai_detection:
server_endpoint: "http://localhost:8080/process/text"
input:
entities: []
output:
entities: []
retrieval:
entities: []

For more information, refer to Private AI Integration.

Fiddler Guardrails
rails:
config:
fiddler:
fiddler_endpoint: "http://localhost:8080/process/text"
safety_threshold: 0.1
faithfulness_threshold: 0.05

For more information, refer to Fiddler Guardrails Integration.

Guardrails AI
rails:
config:
guardrails_ai:
input:
validators:
- name: toxic_language
parameters:
threshold: 0.5
metadata: {}
output:
validators:
- name: pii
parameters: {}

For more information, refer to Guardrails AI Integration.


Prompts Configuration

Define prompts for LLM tasks.

prompts:
- task: self_check_input
content: |
Your task is to check if the user input is safe.
User input: {{ user_input }}
Answer [Yes/No]:
output_parser: null
max_length: 16000
max_tokens: null
mode: standard
stop: null
models: null # Restrict to specific engines/models
AttributeTypeDefaultDescription
prompts.contentstringPrompt template (mutually exclusive with messages)
prompts.max_lengthinteger16000Maximum prompt length (characters)
prompts.max_tokensintegernullMaximum response tokens
prompts.messageslistChat messages (mutually exclusive with content)
prompts.modestring"standard"Prompting mode
prompts.modelslistnullRestrict to engines/models (e.g., ["openai", "nim/llama-3.1"])
prompts.output_parserstringnullOutput parser name
prompts.stoplistnullStop tokens
prompts.taskstring✓Task identifier

Available Tasks

The following table lists all available tasks you can specify to prompts.task.

TaskDescription
generalGeneral response generation (no dialog rails)
generate_bot_messageGenerate bot response
generate_next_stepsDetermine next conversation step
generate_user_intentGenerate canonical user intent
self_check_factsVerify factual accuracy of responses
self_check_hallucinationDetect hallucinations in responses
self_check_inputCheck if user input complies with policy
self_check_outputCheck if bot output complies with policy

Available Prompt Message Types

The following table lists all available message types you can specify to prompts.messages.type.

TypeDescription
assistantAssistant/bot message content
botAlias for assistant
systemSystem-level instructions
userUser message content

Other Configuration Options

Instructions

instructions:
- type: general
content: |
You are a helpful assistant.

Sample Conversation

sample_conversation: |
user "Hello there!"
express greeting
bot express greeting
"Hello! How can I assist you today?"
user "What can you do for me?"
ask about capabilities
bot respond about capabilities
"As an AI assistant, I can help you with a wide range of tasks."

Knowledge Base

knowledge_base:
folder: kb
embedding_search_provider:
name: default
parameters: {}
cache:
enabled: false

Core Settings

core:
embedding_search_provider:
name: default
parameters: {}

Tracing

tracing:
enabled: false
adapters:
- name: FileSystem
span_format: opentelemetry
enable_content_capture: false

Streaming

v0.20.0
The top-level `streaming` field is a boolean that is no longer required. Use the `stream_async()` method directly instead. For output rail streaming configuration, see [Output Streaming Configuration](#output-streaming-configuration).
streaming: false

Hooks Server Tool Policy

The top-level hooks_server section sets the tool policy of the Guardrails hooks server, which applies it to each tool call before any rail runs. Only the hooks server reads it: the Guardrails API server and the Python API ignore it.

hooks_server:
allowed_tools:
- Bash
- name: search
arguments: [query]
blocked_tools:
- WebFetch
- name: Bash
arguments: [run_in_background]
allow_unlisted_tools: false
FieldDefaultDescription
allowed_tools[]The tools a call may use. A name allows any arguments; a {name, arguments} entry allows only the listed arguments.
blocked_tools[]The tools a call may not use. A name blocks every call; a {name, arguments} entry blocks a call that passes any of the listed arguments.
allow_unlisted_toolstrueWhether a call to a tool that allowed_tools does not name is allowed. Set it whenever allowed_tools lists a tool: the hooks server refuses to start an allowlist that leaves it unset.

The configuration fails to load when a list names a tool twice, when allowed_tools allows a tool that blocked_tools blocks by name, when a {name, arguments} entry has an empty arguments list or names an argument twice, or when the section or an entry has an unknown key.

Set hooks_server in one file. When more than one file of a configuration, or a configuration it imports through import_paths, sets it, the configuration fails to load, naming the file where the second section was found. Likewise, combining two configurations with + raises ValueError when both set a hooks_server section. Keeping the policy in one file means no other file can drop its restrictions or widen its allowlist. For how the hooks server applies the policy, refer to Run the Hooks Server: The Tool Policy.

Import Paths

import_paths:
- path/to/shared/config

Complete Example

The following config.yml example defines a main language model, a dedicated content safety model, and an embeddings model. It enables input and output content safety checks with a prompt for each check and configures output rail streaming. The main model uses NVIDIA-hosted inference and requires NVIDIA_API_KEY; start the content safety model at http://localhost:8123/v1 before running the configuration. The port avoids a conflict with the Guardrails server’s default port, 8000. For deployment steps, refer to Nemotron Safety Guard Deployment.

The prompts are the NemoGuard content safety prompts from the content safety example, which match the response format that the configured parsers expect.

models:
# Main application LLM
- type: main
engine: nim
model: meta/llama-3.1-70b-instruct
parameters:
temperature: 0.7
# Content safety model
- type: content_safety
engine: nim
parameters:
base_url: "http://localhost:8123/v1"
model_name: "nvidia/llama-3.1-nemotron-safety-guard-8b-v3"
# Embeddings
- type: embeddings
engine: FastEmbed
model: all-MiniLM-L6-v2
rails:
input:
flows:
- content safety check input $model=content_safety
output:
flows:
- content safety check output $model=content_safety
streaming:
enabled: true
prompts:
- task: content_safety_check_input $model=content_safety
content: |
Task: Check if there is unsafe content in the user or agent messages in the below conversation in accordance with our safety policy with the below categories. Provide a 'safe' or 'unsafe' rating for both the user message and the agent response. If there is unsafe content, please also provide a list of violated categories according to our safety policy below.
<BEGIN UNSAFE CONTENT CATEGORIES>
S1: Violence.
S2: Sexual.
S3: Criminal Planning/Confessions.
S4: Guns and Illegal Weapons.
S5: Controlled/Regulated Substances.
S6: Suicide and Self Harm.
S7: Sexual (minor).
S8: Hate/Identity Hate.
S9: PII/Privacy.
S10: Harassment.
S11: Threat.
S12: Profanity.
S13: Needs Caution.
S14: Other.
S15: Manipulation.
S16: Fraud/Deception.
S17: Malware.
S18: High Risk Gov Decision Making.
S19: Political/Misinformation/Conspiracy.
S20: Copyright/Trademark/Plagiarism.
S21: Unauthorized Advice.
S22: Illegal Activity.
S23: Immoral/Unethical.
<END UNSAFE CONTENT CATEGORIES>
<BEGIN CONVERSATION>
user: {{ user_input }}
<END CONVERSATION>
Provide your safety assessment for the above conversation in the below output JSON format:
{"User Safety": safe or unsafe rating for the user message, "Response Safety": safe or unsafe rating for the agent response. Omit if no agent response present. "Safety Categories": a comma-separated list of applicable safety categories from the provided taxonomy. Omit if all safe.}
Do not include anything other than the output JSON in your response.
Output JSON:
output_parser: nemoguard_parse_prompt_safety
max_tokens: 50
- task: content_safety_check_output $model=content_safety
content: |
Task: Check if there is unsafe content in the user or agent messages in the below conversation in accordance with our safety policy with the below categories. Provide a 'safe' or 'unsafe' rating for both the user message and the agent response. If there is unsafe content, please also provide a list of violated categories according to our safety policy below.
<BEGIN UNSAFE CONTENT CATEGORIES>
S1: Violence.
S2: Sexual.
S3: Criminal Planning/Confessions.
S4: Guns and Illegal Weapons.
S5: Controlled/Regulated Substances.
S6: Suicide and Self Harm.
S7: Sexual (minor).
S8: Hate/Identity Hate.
S9: PII/Privacy.
S10: Harassment.
S11: Threat.
S12: Profanity.
S13: Needs Caution.
S14: Other.
S15: Manipulation.
S16: Fraud/Deception.
S17: Malware.
S18: High Risk Gov Decision Making.
S19: Political/Misinformation/Conspiracy.
S20: Copyright/Trademark/Plagiarism.
S21: Unauthorized Advice.
S22: Illegal Activity.
S23: Immoral/Unethical.
<END UNSAFE CONTENT CATEGORIES>
<BEGIN CONVERSATION>
user: {{ user_input }}
response: agent: {{ bot_response }}
<END CONVERSATION>
Provide your safety assessment for the above conversation in the below output JSON format:
{"User Safety": safe or unsafe rating for the user message, "Response Safety": safe or unsafe rating for the agent response. Omit if no agent response present. "Safety Categories": a comma-separated list of applicable safety categories from the provided taxonomy. Omit if all safe.}
Do not include anything other than the output JSON in your response.
Output JSON:
output_parser: nemoguard_parse_response_safety
max_tokens: 50
instructions:
- type: general
content: |
You are a helpful, harmless, and honest assistant.