Guardrails Configuration

This section describes how to configure guardrails in the config.yml file to control LLM behavior.

The rails Key

The rails key defines which guardrails are active and their configuration options. Rails are organized into five categories based on when they trigger during the guardrails process.

Rail Categories

The following table summarizes the different rail categories and their trigger points.

The following diagram shows the guardrails process described in the table above in detail.

Basic Configuration

1
rails:
2
    input:
3
        flows:
4
            - self check input
5
            - jailbreak detection heuristics
6
            - mask sensitive data on input
7
8
    output:
9
        flows:
10
            - self check output
11
            - self check facts
12
            - check output sensitive data
13
14
    retrieval:
15
        flows:
16
            - check retrieval sensitive data

Input Rails

Input rails process user messages before they reach the LLM:

1
rails:
2
    input:
3
        flows:
4
            - self check input # LLM-based input validation
5
            - jailbreak detection heuristics # Jailbreak detection
6
            - mask sensitive data on input # PII masking

For a complete list of available input flows, refer to the YAML Schema Reference: Input Rails.

Output Rails

Output rails process LLM responses before returning to users:

1
rails:
2
    output:
3
        flows:
4
            - self check output # LLM-based output validation
5
            - self check facts # Fact verification
6
            - self check hallucination # Hallucination detection
7
            - mask sensitive data on output # PII masking

For a complete list of available output flows, refer to the YAML Schema Reference: Output Rails.

Retrieval Rails

Retrieval rails process chunks retrieved from the knowledge base:

1
rails:
2
    retrieval:
3
        flows:
4
            - check retrieval sensitive data

For a complete list of available retrieval flows, refer to the YAML Schema Reference: Retrieval Rails.

Dialog Rails

Dialog rails control conversation flow after user intent is determined:

1
rails:
2
    dialog:
3
        single_call:
4
            enabled: false
5
            fallback_to_multiple_calls: true
6
7
        user_messages:
8
            embeddings_only: false

For a complete list of available dialog flows, refer to the YAML Schema Reference: Dialog Rails.

Execution Rails

Execution rails control custom action and tool invocations:

1
rails:
2
    execution:
3
        flows:
4
            - check tool input
5
            - check tool output

Rail-Specific Configuration

Configure options for specific rails using the config key:

1
rails:
2
    config:
3
        # Sensitive data detection settings
4
        sensitive_data_detection:
5
            input:
6
                entities:
7
                    - PERSON
8
                    - EMAIL_ADDRESS
9
                    - PHONE_NUMBER
10
            output:
11
                entities:
12
                    - PERSON
13
                    - EMAIL_ADDRESS
14
15
        # Jailbreak detection settings
16
        jailbreak_detection:
17
            length_per_perplexity_threshold: 89.79
18
            prefix_suffix_perplexity_threshold: 1845.65
19
20
        # Fact-checking settings
21
        fact_checking:
22
            parameters:
23
                endpoint: "http://localhost:5000"

YAML Schema

Complete guardrails configuration example:

1
rails:
2
    # Input validation
3
    input:
4
        flows:
5
            - self check input
6
            - jailbreak detection heuristics
7
            - mask sensitive data on input
8
9
    # Output validation
10
    output:
11
        flows:
12
            - self check output
13
            - self check facts
14
15
    # Retrieval processing
16
    retrieval:
17
        flows:
18
            - check retrieval sensitive data
19
20
    # Dialog behavior
21
    dialog:
22
        single_call:
23
            enabled: false
24
25
    # Rail-specific settings
26
    config:
27
        sensitive_data_detection:
28
            input:
29
                entities:
30
                    - PERSON
31
                    - EMAIL_ADDRESS
32
                    - CREDIT_CARD
33
            output:
34
                entities:
35
                    - PERSON
36
                    - EMAIL_ADDRESS

Parallel Execution of Input and Output Rails

You can configure input and output rails to run in parallel. This can improve latency and throughput.

IORails Engine

The IORails engine is an optimized execution engine that runs NemoGuard input and output rails in parallel with dedicated model management. The IORails engine is an opt-in feature. By default, the NeMo Guardrails library uses the LLMRails engine.

Supported Flows

The IORails engine supports the following flows:

• content safety check input / content safety check output
• topic safety check input
• jailbreak detection model

When IORails is enabled and the configuration uses only these flows, the engine runs them in parallel. Configurations that include custom flows, dialog rails, or other unsupported flows silently fall back to the LLMRails engine and emit a warning. Pass require_iorails=True to Guardrails(...) to raise a ValueError at initialization instead.

Enabling IORails

To enable the IORails engine, set the NEMO_GUARDRAILS_IORAILS_ENGINE environment variable to 1:

$
NEMO_GUARDRAILS_IORAILS_ENGINE=1 nemoguardrails chat --config examples/configs/content_safety

When using the Python API, import the Guardrails class directly and pass use_iorails=True:

1
from nemoguardrails import Guardrails, RailsConfig
2
3
config = RailsConfig.from_path("./config")
4
# require_iorails=True ensures the engine is IORails (raises on fallback), so
5
# parallel execution is actually in effect — the whole reason for opting in here.
6
guardrails = Guardrails(config, use_iorails=True, require_iorails=True)

YAML-Based Parallel Execution

You can also configure existing LLMRails flows to run in parallel using the parallel: True option in the config.yml file. This approach works with any flow type and does not require the IORails engine.

When to Use

Use YAML-based parallel execution:

• For I/O-bound rails such as external API calls to LLMs or third-party integrations.
• If you have two or more independent input or output rails without shared state dependencies.
• In production environments where response latency affects user experience and business metrics.

When Not to Use

Avoid parallel execution:

• For CPU-bound rails; it might not improve performance and can introduce overhead.
• During development and testing for debugging and simpler workflows.

Configuration Example

To enable parallel execution, set parallel: True in the rails.input and rails.output sections in the config.yml file.

The following is an example configuration for parallel rails using models from NVIDIA Cloud Functions (NVCF). When you use NVCF models, make sure that you export NVIDIA_API_KEY to access those models.

Save the following code snippet to config.yml. Download prompts.yaml and put this in the same directory as the config.yml.

1
models:
2
    - type: main
3
      engine: nim
4
      model: meta/llama-3.1-70b-instruct
5
    - type: content_safety
6
      engine: nim
7
      model: nvidia/llama-3.1-nemoguard-8b-content-safety
8
    - type: topic_control
9
      engine: nim
10
      model: nvidia/llama-3.1-nemoguard-8b-topic-control
11
12
rails:
13
    input:
14
        parallel: True
15
        flows:
16
            - content safety check input $model=content_safety
17
            - topic safety check input $model=topic_control
18
    output:
19
        parallel: True
20
        flows:
21
            - content safety check output $model=content_safety
22
            - self check output
23
        streaming:
24
            enabled: True
25
            chunk_size: 200
26
            context_size: 50
27
            stream_first: True

Speculative Generation

Speculative generation runs input-rail and main LLM response generation in parallel, rather than sequentially. If response generation takes longer than the input-rail latency, this hides the latency of the input-rail check. The tradeoff is that the main LLM will begin generating a response for unsafe requests, with a corresponding token cost. However, responses are always checked by output rails before being returned to the client so no unsafe responses will be seen.

When to use Speculative Generation

In many applications, safe requests are much more likely than unsafe requests. Speculative generation takes advantage of this by assuming all requests are safe for generation. Assuming a 2% rate of unsafe requests, the remaining 98% of safe requests will hide the input-rail latency by running in parallel with response generation. The cost of this latency saving is that tokens for the 2% of unsafe requests will be generated and then discarded. To decide whether Speculative Generation makes sense for your use-case, explore the unsafe request rate and potential latency savings.

How Speculative Generation Works

Without speculative generation, the IORails engine runs the input rails first and only starts the main LLM call once the input is determined to be safe:

1. Run input rails on the user message. If the input is unsafe, return the refusal message and stop.
2. If the input is safe, generate a response from the main LLM.
3. Run output rails on the LLM response. If the output is unsafe, return the refusal message and stop.
4. Return the response.

With speculative generation enabled, the input rails and the main LLM call start at the same time and race to completion:

1. Start the input rails and the main LLM call in parallel.
2. Wait for whichever finishes first, then resolve the race:
   • If the input rails finish first and the input is unsafe, cancel the LLM call and return the refusal message.
   • If the input rails finish first and the input is safe, wait for the LLM call to finish.
   • If the LLM call finishes first, wait for the input-rail verdict; discard the response and return the refusal message if the input is unsafe.
3. Run output rails on the LLM response.
4. Return the response, or the refusal message if output rails blocked it.

The engine handles three outcomes:

Output rails always run after the main LLM completes. Speculative generation does not change the output-rail path.

Configuration Example

To enable speculative generation, set speculative_generation: True under rails.input in the config.yml file. Speculative generation requires the IORails engine; see IORails Engine for how to enable it.

1
models:
2
    - type: main
3
      engine: nim
4
      model: meta/llama-3.1-70b-instruct
5
    - type: content_safety
6
      engine: nim
7
      model: nvidia/llama-3.1-nemoguard-8b-content-safety
8
9
rails:
10
    input:
11
        speculative_generation: True
12
        flows:
13
            - content safety check input $model=content_safety
14
    output:
15
        flows:
16
            - content safety check output $model=content_safety

speculative_generation and parallel can be combined. Input rails will run in parallel with each other and concurrently with the main LLM call.