About NeMo Guardrails Library

Supported LLMs

View as Markdown

The NVIDIA NeMo Guardrails library supports a wide range of LLM providers and models, including base, instruct-tuned, and reasoning models. You can serve these models locally on the same machine as the library or at a remote network endpoint. This flexible approach supports applications from edge deployments on resource-constrained devices to horizontally scalable backend clusters.

LLM Types

Integrating the library improves the safety and security of an application LLM, which generates responses to the end user. The library can also use the same application LLM to run guardrails, which simplifies deployments and reduces onboarding friction. Two examples are self-check rails and dialog rails. Self-check rails use the application LLM to decide whether a user request or LLM response is safe. Dialog rails use the application LLM to guide the user through a predefined conversational flow.

The library can also call models for a specific guardrail on behalf of the client. Guardrail-specific models let you use smaller fine-tuned models that specialize in the guardrails task. For example, the NVIDIA NemoGuard collection of models includes content-safety, topic-control, and jailbreak-detect models. You can access these models on build.nvidia.com for rapid prototyping or on NGC Catalog for deployment with NIM Docker containers.

Inference Providers

Each engine is served by a framework that manages the underlying HTTP or SDK calls. The library ships with a built-in framework that talks to OpenAI-compatible endpoints over httpx with no LangChain dependency. For engines whose API is not OpenAI-compatible, opt into the LangChain framework by setting NEMOGUARDRAILS_LLM_FRAMEWORK=langchain and installing the matching langchain-<provider> package. To add a custom framework, implement the LLMFramework protocol from nemoguardrails.types.

html
1<button type="button" class="table-expand-button" data-table-title="Inference Providers">
2  <span aria-hidden="true" class="table-expand-button__icon">&#x26F6;</span>
3  Expand table
4</button>
EngineFrameworkStreamingTool callsReasoning modelsNotes
anthropicLangChain (opt-in)yesyeswrapper-dependentRequires pip install langchain langchain-anthropic.
azure, azure_openaiBuilt-inyesyesyesNative support for key-based Azure OpenAI authentication. Set parameters.azure_endpoint or parameters.base_url to the resource endpoint, plus azure_deployment and api_version; for Azure AD or token-based authentication, use manual engine: openai configuration or opt into LangChain.
cohereLangChain (opt-in)yesyesn/aRequires pip install langchain langchain-cohere.
google_genaiLangChain (opt-in)yesyesn/aRequires pip install langchain langchain-google-genai.
huggingface_endpointLangChain (opt-in)variesvariesvariesDefault text-generation schema. If your endpoint exposes /v1/chat/completions, prefer engine: openai with parameters.base_url instead.
huggingface_pipeline, huggingface_hub, trt_llm, self_hostedLangChain (opt-in)variesvariesvariesIn-process pipelines and LangChain wrappers without a native HTTP path.
nimBuilt-inyesyesyesDefault base URL https://integrate.api.nvidia.com/v1.
nvidia_ai_endpointsBuilt-inyesyesyesAlias for nim.
ollamaBuilt-inyesyesyes (where supported)Default base URL http://localhost:11434/v1.
openaiBuilt-inyesyesyesOpenAI public API or any OpenAI-compatible endpoint using parameters.base_url. For vLLM, TGI, OpenRouter, Together.ai, Fireworks.ai, Groq, DeepSeek, llama.cpp, NVIDIA Nemotron, and similar providers, use engine: openai with parameters.base_url and parameters.api_key.
vertexaiLangChain (opt-in)yesyesn/aRequires pip install langchain langchain-google-vertexai.
vllm_openai, deepseekLangChain (opt-in)yesyesyesLegacy LangChain provider engines. They continue to work when you opt into LangChain. For new configurations, use engine: openai with parameters.base_url when the wire protocol is OpenAI-compatible.
<provider_name>LangChain (opt-in)variesvariesvariesAny community provider exposed through LangChain’s chat-model integrations. Use the bare provider name as the engine name.

For migration recipes between the built-in path and the LangChain path, see Migrating to 0.22.

LangChain-Backed Providers

The library supports LLM providers from the LangChain Community, including text completion and chat completion providers. Refer to Chat model integrations in the LangChain documentation. You can also use the nemoguardrails find-providers CLI command to discover available providers.

Embedding Model Providers

The library uses embedding models for vector similarity search in dialog rails, embeddings_only intent matching, and knowledge base retrieval. The following table lists the supported embedding model providers and their corresponding engine names.

ProviderEngineNotes
NVIDIA NIMnimNVIDIA NIM microservices
NVIDIA AI Endpointsnvidia_ai_endpointsAlias for nim
FastEmbedfastembedFastEmbed embedding model provider
OpenAIopenaiOpenAI embedding model provider
Azure OpenAIazureAzure OpenAI embedding model provider
CoherecohereCohere embedding model provider
SentenceTransformerssentence_transformersSentenceTransformers embedding model provider
GooglegoogleGoogle embedding model provider

For more information about configuring embedding providers, refer to Embedding Search Providers.