Using LLMs hosted on NVIDIA API Catalog

This guide teaches you how to use NeMo Guardrails with LLMs hosted on NVIDIA API Catalog. It uses the ABC Bot configuration and with the meta/llama-3.1-70b-instruct model. Similarly, you can use meta/llama-3.1-405b-instruct, meta/llama-3.1-8b-instruct or any other AI Foundation Model.

Prerequisites

Before you begin, ensure you have the following prerequisites in place:

  1. Install the langchain-nvidia-ai-endpoints package:

pip install -U --quiet langchain-nvidia-ai-endpoints

  1. An NVIDIA NGC account to access AI Foundation Models. To create a free account go to NVIDIA NGC website.

  2. An API key from NVIDIA API Catalog:

    • Generate an API key by navigating to the AI Foundation Models section on the NVIDIA NGC website, selecting a model with an API endpoint, and generating an API key. You can use this API key for all models available in the NVIDIA API Catalog.

    • Export the NVIDIA API key as an environment variable:

export NVIDIA_API_KEY=$NVIDIA_API_KEY # Replace with your own key

  1. If you’re running this inside a notebook, patch the AsyncIO loop.

import nest_asyncio

nest_asyncio.apply()

Configuration

To get started, copy the ABC bot configuration into a subdirectory called config:

cp -r ../../../../examples/bots/abc config

Update the models section of the config.yml file to the desired model supported by NVIDIA API Catalog:

...
models:
  - type: main
    engine: nvidia_ai_endpoints
    model: meta/llama-3.1-70b-instruct
...

Usage

Load the guardrail configuration:

from nemoguardrails import LLMRails, RailsConfig

config = RailsConfig.from_path("./config")
rails = LLMRails(config)

Test that it works:

response = rails.generate(messages=[
{
    "role": "user",
    "content": "How many vacation days do I have per year?"
}])
print(response['content'])

According to our company policy, you are eligible for 20 days of vacation per year, accrued monthly.

You can see that the bot responds correctly.

Conclusion

In this guide, you learned how to connect a NeMo Guardrails configuration to an NVIDIA API Catalog LLM model. This guide uses meta/llama-3.1-70b-instruct, however, you can connect any other model by following the same steps.