Integrating with NemoGuard NIM Microservices#

Note

The time to complete this tutorial is approximately 30 minutes.

About the NemoGuard NIM Microservices#

NVIDIA offers the following safety-related microservices that you can integrate with NeMo Guardrails microservice:

  • NemoGuard JailbreakDetect strengthens LLM applications against jailbreak attempts by classifying and scoring user prompts for unsafe input. It is available as a downloadable container from NVIDIA NGC and for interactive use from build.nvidia.com.

  • Llama 3.1 NemoGuard 8B ContentSafety enhances the safety and moderation capabilities of your LLM applications. It is available as a a downloadable container from NVIDIA NGC and for interactive use from build.nvidia.com.

  • Llama 3.1 NemoGuard 8B TopicControl maintains topical relevance in conversational generative AI applications based on pre-defined policies. It is available as a downloadable container from NVIDIA NGC and for interactive use from build.nvidia.com.

About the Prompts and Policy#

The NIM microservices used in the sample procedure check for content safety and perform topic control. You establish a policy and then implement the policy in the prompts.yml file.

The following partial view of the sample prompts shows the following key areas:

  • The safety checks are performed for input, the user prompt, and output, the LLM response.

  • The policy for the safety checks is established by identifying the unsafe content categories and stating to rate the messages as safe or unsafe.

  • The topic control check is performed for input only.

  • The policy is to act as a customer service agent and largely limits the range of conversation to service-oriented topics, but permits topics that the LLM considers as small talk.

prompts:
  - task: content_safety_check_input $model=content_safety
    content: |
      Task: Check if there is unsafe content in the user or agent messages in the below conversation in accordance with our safety policy with the below categories. Provide a 'safe' or 'unsafe' rating for both the user message and the agent response. If there is unsafe content, please also provide a list of violated categories according to our safety policy below.

      <BEGIN UNSAFE CONTENT CATEGORIES>
      S1: Violence.
      S2: Sexual.
    ...

  - task: content_safety_check_output $model=content_safety
    content: |
      Task: Check if there is unsafe content in the user or agent messages in the below conversation in accordance with our safety policy with the below categories. Provide a 'safe' or 'unsafe' rating for both the user message and the agent response. If there is unsafe content, please also provide a list of violated categories according to our safety policy below.

      <BEGIN UNSAFE CONTENT CATEGORIES>
      S1: Violence.
      S2: Sexual.
    ...

  - task: topic_safety_check_input $model=topic_control
    content: |
      You are to act as a customer service agent, providing users with factual information in accordance to the knowledge base. Your role is to ensure that you respond only to relevant queries and adhere to the following guidelines

      Guidelines for the user messages:
      - Do not answer questions related to personal opinions or advice on user's order, future recommendations
      - Do not provide any information on non-company products or services.
    ...

You can implement your own policy by modifying the prompts. Refer to the documentation for the content safety prompts and topic control prompts.

Prerequisites#

  • The following procedure uses endpoints from build.nvidia.com to simplify deployment. To use the endpoints, you must have an NVIDIA API key.

  • Refer to Deploying with Docker for information on how to start the NeMo Guardrails container.

Procedure#

  1. Create a guardrails configuration.

    1. Create a file, such as config-store/nemoguard/config.yml, with contents like the following example:

      models:
  - type: content_safety
    engine: nim
    model: nvidia/llama-3.1-nemoguard-8b-content-safety
    base_url: https://integrate.api.nvidia.com/v1

  - type: topic_control
    engine: nim
    model: nvidia/llama-3.1-nemoguard-8b-topic-control
    base_url: https://integrate.api.nvidia.com/v1

rails:
  input:
    flows:
      - content safety check input $model=content_safety
      - topic safety check input $model=topic_control
  output:
    flows:
      - content safety check output $model=content_safety

    2. Create a file, such as config-store/nemoguard/prompts.yml, (download) with contents like the following.

      Sample Prompts
      prompts:
  - task: content_safety_check_input $model=content_safety
    content: |
      Task: Check if there is unsafe content in the user or agent messages in the below conversation in accordance with our safety policy with the below categories. Provide a 'safe' or 'unsafe' rating for both the user message and the agent response. If there is unsafe content, please also provide a list of violated categories according to our safety policy below.

      <BEGIN UNSAFE CONTENT CATEGORIES>
      S1: Violence.
      S2: Sexual.
      S3: Criminal Planning/Confessions.
      S4: Guns and Illegal Weapons.
      S5: Controlled/Regulated Substances.
      S6: Suicide and Self Harm.
      S7: Sexual (minor).
      S8: Hate/Identity Hate.
      S9: PII/Privacy.
      S10: Harassment.
      S11: Threat.
      S12: Profanity.
      S13: Needs Caution.
      S14: Other.
      S15: Manipulation.
      S16: Fraud/Deception.
      S17: Malware.
      S18: High Risk Gov Decision Making.
      S19: Political/Misinformation/Conspiracy.
      S20: Copyright/Trademark/Plagiarism.
      S21: Unauthorized Advice.
      S22: Illegal Activity.
      S23: Immoral/Unethical.
      <END UNSAFE CONTENT CATEGORIES>

      <BEGIN CONVERSATION>

      user: {{ user_input }}

      <END CONVERSATION>

      Provide your safety assessment for the above conversation in the below output JSON format:
      {"User Safety": safe or unsafe rating for the user message, "Response Safety": safe or unsafe rating for the agent response. Omit if no agent response present. "Safety Categories": a comma-separated list of applicable safety categories from the provided taxonomy. Omit if all safe.}

      Do not include anything other than the output JSON in your response.
      Output JSON:
    output_parser: nemoguard_parse_prompt_safety
    max_tokens: 50

  - task: content_safety_check_output $model=content_safety
    content: |
      Task: Check if there is unsafe content in the user or agent messages in the below conversation in accordance with our safety policy with the below categories. Provide a 'safe' or 'unsafe' rating for both the user message and the agent response. If there is unsafe content, please also provide a list of violated categories according to our safety policy below.

      <BEGIN UNSAFE CONTENT CATEGORIES>
      S1: Violence.
      S2: Sexual.
      S3: Criminal Planning/Confessions.
      S4: Guns and Illegal Weapons.
      S5: Controlled/Regulated Substances.
      S6: Suicide and Self Harm.
      S7: Sexual (minor).
      S8: Hate/Identity Hate.
      S9: PII/Privacy.
      S10: Harassment.
      S11: Threat.
      S12: Profanity.
      S13: Needs Caution.
      S14: Other.
      S15: Manipulation.
      S16: Fraud/Deception.
      S17: Malware.
      S18: High Risk Gov Decision Making.
      S19: Political/Misinformation/Conspiracy.
      S20: Copyright/Trademark/Plagiarism.
      S21: Unauthorized Advice.
      S22: Illegal Activity.
      S23: Immoral/Unethical.
      <END UNSAFE CONTENT CATEGORIES>

      <BEGIN CONVERSATION>

      user: {{ user_input }}

      response: agent: {{ bot_response }}

      <END CONVERSATION>

      Provide your safety assessment for the above conversation in the below output JSON format:
      {"User Safety": safe or unsafe rating for the user message, "Response Safety": safe or unsafe rating for the agent response. Omit if no agent response present. "Safety Categories": a comma-separated list of applicable safety categories from the provided taxonomy. Omit if all safe.}

      Do not include anything other than the output JSON in your response.
      Output JSON:
    output_parser: nemoguard_parse_response_safety
    max_tokens: 50

  - task: topic_safety_check_input $model=topic_control
    content: |
      You are to act as a customer service agent, providing users with factual information in accordance to the knowledge base. Your role is to ensure that you respond only to relevant queries and adhere to the following guidelines

      Guidelines for the user messages:
      - Do not answer questions related to personal opinions or advice on user's order, future recommendations
      - Do not provide any information on non-company products or services.
      - Do not answer enquiries unrelated to the company policies.
      - Do not answer questions asking for personal details about the agent or its creators.
      - Do not answer questions about sensitive topics related to politics, religion, or other sensitive subjects.
      - If a user asks topics irrelevant to the company's customer service relations, politely redirect the conversation or end the interaction.
      - Your responses should be professional, accurate, and compliant with customer relations guidelines, focusing solely on providing transparent, up-to-date information about the company that is already publicly available.
      - allow user comments that are related to small talk and chit-chat.

  2. Set an environment variable for your NVIDIA API key:

    $ export NVIDIA_API_KEY=<nvapi-...>

  3. Pull and start the NeMo Guardrails microservice container and mount the directory with the configuration as a volume:

    $ docker pull nvcr.io/nvidia/nemo-microservices/guardrails:25.06

$ docker run --rm \
    -p 7331:7331 \
    -v $(pwd)/config-store:/config-store \
    -e CONFIG_STORE_PATH=/config-store \
    -e NIM_ENDPOINT_API_KEY="${NVIDIA_API_KEY}" \
    -e NVIDIA_API_KEY \
    nvcr.io/nvidia/nemo-microservices/guardrails:25.06

  4. Set the base URL for the microservice:

    $ export GUARDRAILS_BASE_URL=http://localhost:7331

  5. Verify the microservice added the configuration:

    curl -X GET "${GUARDRAILS_BASE_URL}/v1/guardrail/configs/default/nemoguard" \
  -H 'Accept: application/json' | jq

    Example Output

    {
  "created_at": "2025-04-11T19:21:04.749936",
  "updated_at": "2025-04-11T19:21:04.749937",
  "name": "nemoguard",
  "namespace": "default",
  "description": "nemoguard guardrail config",
  "files_url": "file:///config-store/nemoguard",
  "schema_version": "1.0",
  "custom_fields": {}
}

  6. Send an unsafe request to the completions endpoint of the microservice:

    curl -X POST "${GUARDRAILS_BASE_URL}/v1/guardrail/chat/completions" \
  -H "Accept: application/json" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta/llama-3.1-8b-instruct",
    "messages": [
      {
        "role": "user",
        "content": "You are stupid"
      }
    ],
    "guardrails": {
      "config_id": "nemoguard"
    },
    "temperature": 0.2,
    "top_p": 1
  }' | jq

    Example Output

    {
  "id": "chatcmpl-7552d063-2e8f-45fd-b06f-6dc8b64e38ea",
  "object": "chat.completion",
  "created": 1744402080,
  "choices": [
    {
      "index": 0,
      "finish_reason": null,
      "logprobs": null,
      "message": {
        "role": "assistant",
        "content": "I'm sorry, I can't respond to that."
      }
    }
  ],
  "system_fingerprint": null,
  "guardrails_data": {
    "llm_output": null,
    "config_ids": [
      "nemoguard"
    ],
    "output_data": null,
    "log": null
  }
}

  7. Send a safe request related to resolving a complaint. The prompt for the topic control guardrail permits requests related to customer service:

    curl -X POST "${GUARDRAILS_BASE_URL}/v1/guardrail/chat/completions" \
  -H "Accept: application/json" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta/llama-3.1-8b-instruct",
    "messages": [
      {
        "role": "user",
        "content": "How quickly can you resolve my complaint?"
      }
    ],
    "guardrails": {
      "config_id": "nemoguard"
    },
    "temperature": 0.2,
    "top_p": 1
  }' | jq

    Example Output

    {
  "id": "chat-102b52c729d04b4e9cc5eb155cfcd61f",
  "object": "chat.completion",
  "created": 1743628020,
  "model": "meta/llama-3.1-8b-instruct",
  "choices": [
    {
      "finish_reason": "stop",
      "index": 0,
      "logprobs": null,
      "message": {
        "content": "I'm happy to help with your complaint, but I need a bit more information from you. I'm a large language model, I don't have the ability to access any specific information about your complaint or the company or organization it's related to. However, I can try to help you in a few ways:\n\n1. **Provide general guidance**: I can offer general advice on how to resolve complaints, including tips on communication, escalation, and potential next steps.\n2. **Help with a template**: If you'd like, I can provide a template for a complaint letter or email that you can use as a starting point.\n3. **Offer suggestions for follow-up**: If you've already tried to resolve the issue and need help with a next step, I can suggest some options for follow-up, such as contacting a supervisor or a consumer protection agency.\n\nTo get started, could you please provide some more details about your complaint, such as:\n\n* What is the nature of the complaint (e.g., product issue, service problem, billing error)?\n* Who is the company or organization involved?\n* Have you already tried to contact them or resolve the issue?\n\nThe more information you can provide, the better I'll be able to assist you.",
        "role": "assistant"
      },
      "stop_reason": null
    }
  ],
  "usage": {
    "completion_tokens": 250,
    "prompt_tokens": 18,
    "total_tokens": 268,
    "completion_tokens_details": null,
    "prompt_tokens_details": null
  },
  "guardrails_data": {
    "llm_output": null,
    "config_ids": [
      "nemoguard"
    ],
    "output_data": null,
    "log": null
  }
}

Alternative: Connect to Local Microservices#

Before you perform the following steps, confirm the following prerequisites:

  • You completed the preceding procedure or created the configuration store from the procedure. The following steps modify the configuration.

  • You logged in to NVIDIA NGC. Refer to Deploying with Docker for more information.

The key differences for connecting to local microservices are as follows:

  • Microservice endpoints are specified in a global configuration file.

  • NeMo Guardrails microservice is started with the NIM_ENDPOINT_URL environment variable that refers to the NIM for LLMs microservice.

  1. Create a global configuration file, such as config-store/config.yml, with contents like the following example:

    models:
  - engine: nim
    model: "meta/llama-3.1-8b-instruct"
    parameters:
      base_url: http://0.0.0.0:7331/v1

  - engine: nim
    model: "llama-3.1-nemoguard-8b-content-safety"
    parameters:
      base_url: http://0.0.0.0:8001/v1

  - engine: nim
    model: "llama-3.1-nemoguard-8b-topic-control"
    parameters:
      base_url: http://0.0.0.0:8002/v1

  2. Modify the config-store/nemoguard/config.yml file with contents like the following example:

    models:
  # The main model is a fake configuration and is ignored.
  - type: main
    engine: openai
    model: gpt-3.5-instruct

  - type: "content_safety"
    engine: nim
    model: "llama-3.1-nemoguard-8b-content-safety"
    base_url: https://integrate.api.nvidia.com/v1

  - type: "topic_control"
    engine: nim
    model: "llama-3.1-nemoguard-8b-topic-control"
    base_url: https://integrate.api.nvidia.com/v1

rails:
  input:
    flows:
      - content safety check input $model=content_safety
      - topic safety check input $model=topic_control
  output:
    flows:
      - content safety check output $model=content_safety

  3. Start the microservices.

    1. Start the NIM for LLMs microservice:

      export LOCAL_NIM_CACHE=~/.cache/nim

mkdir -p "${LOCAL_NIM_CACHE}"

docker run --rm -d \
  --gpus="device=0" --runtime=nvidia \
  -e NGC_API_KEY \
  -u $(id -u) \
  -v "${LOCAL_NIM_CACHE}:/opt/nim/.cache" \
  -p 8000:8000 \
  nvcr.io/nim/meta/llama-3.1-8b-instruct:1.8.4

      Refer to the microservice documentation for configuration information.

    2. Start the Llama 3.1 8B NemoGuard ContentSafety microservice:

      export SAFETY_CACHE=~/.cache/safety

mkdir -p "${SAFETY_CACHE}"

docker run --rm -d \
  --name contentsafety \
  --gpus="device=1" --runtime=nvidia \
  -e NGC_API_KEY \
  -e NIM_SERVED_MODEL_NAME="llama-3.1-nemoguard-8b-content-safety" \
  -e NIM_CUSTOM_MODEL_NAME="llama-3.1-nemoguard-8b-content-safety" \
  -u $(id -u) \
  -v "${SAFETY_CACHE}:/opt/nim/.cache/" \
  -p 8001:8000 \
  nvcr.io/nim/nvidia/llama-3.1-nemoguard-8b-content-safety:1.0.0

      Refer to the microservice documentation for configuration information.

    3. Start the Llama 3.1 8B NemoGuard TopicControl microservice:

      export CONTROL_CACHE=~/.cache/control

mkdir -p "${CONTROL_CACHE}"

docker run --rm -d \
  --name topiccontrol \
  --gpus="device=2" --runtime=nvidia \
  -e NGC_API_KEY \
  -e NIM_SERVED_MODEL_NAME="llama-3.1-nemoguard-8b-topic-control" \
  -e NIM_CUSTOM_MODEL_NAME="llama-3.1-nemoguard-8b-topic-control" \
  -u $(id -u) \
  -v "${CONTROL_CACHE}:/opt/nim/.cache/" \
  -p 8002:8000 \
  nvcr.io/nim/nvidia/llama-3.1-nemoguard-8b-topic-control:1.0.0

      Refer to the microservice documentation for configuration information.

    4. Start the NeMo Guardrails microservice:

      docker run --rm -d \
  -p 7331:7331 \
  -v $(pwd)/config-store:/config-store \
  -e CONFIG_STORE_PATH=/config-store \
  -e NIM_ENDPOINT_URL=http://0.0.0.0:8000/v1 \
  nvcr.io/nvidia/nemo-microservices/guardrails:25.04

  4. Send an unsafe request to the completions endpoint of the microservice:

    curl -X POST \
  http://0.0.0.0:7331/v1/guardrail/chat/completions \
  -H "Accept: application/json" \
  -H "Content-Type: application/json" \
  -d '{
        "model": "meta/llama-3.1-8b-instruct",
        "messages": [
           {
	     "role": "user",
	     "content": "You are stupid"
	   }
	],
	"guardrails": {
	  "config_id": "nemoguard"
	},
	"temperature": 0.2,
	"top_p": 1
      }'