Skip to navigation

NeMo Guardrails supports hallucination detection in RAG systems using Patronus AI’s Lynx model. The model is hosted on Hugging Face and comes in both a 70B parameters (see here) and 8B parameters (see here) variant.

There are three components of hallucination that Lynx checks for:

  • Information in the bot_message is contained in the relevant_chunks
  • There is no extra information in the bot_message that is not in the relevant_chunks
  • The bot_message does not contradict any information in the relevant_chunks

Setup

Because Patronus Lynx is completely open source, you have the flexibility to deploy it in any environment you prefer. An easy method for hosting is to use vLLM.

  1. Get access to Patronus Lynx on HuggingFace. See here for the 70B parameters variant, and here for the 8B parameters variant. The examples below use the 70B parameters model, but there’s no additional configuration to deploy the smaller model, so you can swap the model name references out with 8B.

  2. Log in to Hugging Face

    huggingface-cli login
  3. Install vLLM and spin up a server hosting Patronus Lynx

    pip install vllm
    python -m vllm.entrypoints.openai.api_server --port 5000 --model PatronusAI/Patronus-Lynx-70B-Instruct

    This will launch the vLLM inference server on http://localhost:5000/. You can use the OpenAI API spec to send it a cURL request to make sure it works:

    curl http://localhost:5000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
    "model": "PatronusAI/Patronus-Lynx-70B-Instruct",
    "messages": [
    {"role": "user", "content": "What is a hallucination?"}
    ]
    }'
  4. Create a model called patronus_lynx in your config.yml file, setting the host and port to what you set it as above. If the vLLM is running on a different server from nemoguardrails, you’ll have to replace localhost with the vLLM server’s address. Check out the Patronus Lynx Integration guide for more information.

Ollama

You can also run Patronus Lynx 8B on your personal computer using Ollama!

  1. Install Ollama.

  2. Get access to a GGUF quantized version of Lynx 8B on Huggingface. Check it out here.

  3. Download the gguf model from the repository here. This may take a few minutes.

  4. Create a file called Modelfile with the following contents:

    FROM "./patronus-lynx-8b-instruct-q4_k_m.gguf"
    PARAMETER stop "<|im_start|>"
    PARAMETER stop "<|im_end|>"
    TEMPLATE """
    <|im_start|>system
    {{ .System }}<|im_end|>
    <|im_start|>user
    {{ .Prompt }}<|im_end|>
    <|im_start|>assistant
    """

    Ensure that the FROM field correctly points to the patronus-lynx-8b-instruct-q4_k_m.gguf file you downloaded in Step 3.

  5. Run ollama create patronus-lynx-8b -f Modelfile.

  6. Run ollama run patronus-lynx-8b. You should now be able to chat with patronus-lynx-8b!

  7. Create a model called patronus_lynx in your config.yml file, like this:

    models:
    ...
    - type: patronus_lynx
    engine: ollama
    model: patronus-lynx-8b
    parameters:
    base_url: "http://localhost:11434"

Usage

Here is how to configure your bot to use Patronus Lynx to check for RAG hallucinations in your bot output:

  1. Add a model of type patronus_lynx in config.yml. The example below serves Lynx with vLLM. Because vLLM exposes an OpenAI-compatible API, engine: openai plus parameters.base_url reaches it through NeMo Guardrails’ built-in client with no LangChain dependency. For background, see Migrating to 0.22.

    models:
    ...
    - type: patronus_lynx
    engine: openai
    model: PatronusAI/Patronus-Lynx-70B-Instruct # PatronusAI/Patronus-Lynx-8B-Instruct
    parameters:
    base_url: "http://localhost:5000/v1"
    api_key: EMPTY

    Set api_key: EMPTY (or any non-empty placeholder) when self-hosted vLLM does not enforce auth. If your deployment requires a real token, replace api_key: EMPTY with the literal token value, or omit api_key and set api_key_env_var at the top level of the model entry (not inside parameters:):

    - type: patronus_lynx
    engine: openai
    model: PatronusAI/Patronus-Lynx-70B-Instruct
    api_key_env_var: MY_PATRONUS_LYNX_API_KEY
    parameters:
    base_url: "http://localhost:5000/v1"
  2. Add the guardrail patronus lynx check output hallucination to your output rails in config.yml:

    rails:
    output:
    flows:
    - patronus lynx check output hallucination
  3. Add a prompt for patronus_lynx_check_output_hallucination in the prompts.yml file. We recommend you use the provided prompts.yml:

    prompts:
    - task: patronus_lynx_check_output_hallucination
    content: |
    Given the following QUESTION, DOCUMENT and ANSWER you must analyze ...
    ...

The patronus lynx check output hallucination rail runs the patronus_lynx_check_output_hallucination action, which returns a RailOutcome. The rail blocks the response when is_blocked is true. The metadata contains the hallucination boolean and any reasoning trace. If the model output has no parseable hallucination decision, the action returns a blocking outcome. If relevant_chunks is missing, empty, or not a string, the action logs an error and returns an allowing outcome without checking the response, so make sure your retrieval step provides the chunks.

Here’s the patronus lynx check output hallucination flow, showing how the action is executed:

define bot inform answer unknown
"I don't know the answer to that."
define flow patronus lynx check output hallucination
$patronus_lynx_response = execute patronus_lynx_check_output_hallucination(model_name="patronus_lynx")
$hallucination = $patronus_lynx_response.metadata["hallucination"]
# The Reasoning trace is currently unused, but can be used to modify the bot output
$reasoning = $patronus_lynx_response.metadata["reasoning"]
if $patronus_lynx_response.is_blocked
bot inform answer unknown
stop

Engine Support

patronus lynx check output hallucination runs on LLMRails only.

Lynx judges the response against the retrieved context, so the rail reads relevant_chunks, a conversation value that the Colang retrieval pipeline builds and IORails does not. A configuration that uses this rail routes to LLMRails by default. With require_iorails=True, Guardrails raises ValueError instead of falling back.

For the full per-rail matrix, refer to Rail Engine Support.