Architecture
NeMo Guardrails Gateway adds content inspection around an existing provider HTTP exchange. Provider bindings define the inspection coverage, the NVIDIA NeMo Guardrails library runs configured checks, and the upstream service generates the answer. Its HTTP proxy layer preserves accepted provider JSON bodies and server-sent events (SSE) while applying its path and header forwarding rules.
What Changes in Your Application
Point your provider client’s base URL at the chosen gateway mount, including its API prefix; the quickstart shows the request URL. Continue to send the selected provider operation’s JSON shape. Check the shipped coverage first: a working upstream request can still be outside the gateway’s guarded profile, such as a Chat request containing conversation history or tools.
For a supported Chat request, the binding selects the user message string for
input checks. It constrains the message role and shape, while metadata such as
model remains opaque. Output checks receive the selected assistant text and
that user input as context. The original JSON bytes travel to the provider or
caller only when the relevant validation and checks allow them.
Handle gateway errors as well as upstream errors. A blocked input prevents the inference call; a blocked buffered output withholds the answer. During streaming, a later error can follow already delivered text, so consume the stream’s error format as well as its HTTP status. Refer to caller-visible outcomes.
Choose a Reading Path
Choose the guide that matches your role:
System Context
This view shows the gateway as one software system and its external interactions. Rectangles represent systems or application-owned responsibilities. Solid arrows describe provider traffic; dotted arrows describe optional checking-service calls. Access controls belong to your application or ingress gateway.
The access-controls box represents a responsibility, not a required extra process. It can run in an embedding application’s middleware or an inbound gateway. The launcher binds loopback by default and does not authenticate callers. Checking-service calls serve a separate role from answer generation; the same backend can serve both roles. External services are needed only when your Rails checks use them.
The gateway is included in the NVIDIA NeMo Guardrails library under
nemoguardrails.server.experimental and requires the library’s server extra.
The library’s existing API server remains a separate application. YAML and Python
entry points use the same proxy application factory and lifespan; refer to
application assembly.
Inspection and Forwarding
Inspection uses a selected view of the payload. The gateway retains the original body or event bytes alongside that view and uses check decisions to control their release. It does not serialize an accepted response from a validation model.
This diagram shows a successful upstream response containing guarded text, with output inspection enabled. Rectangles are processing steps or retained data; the diamond is a release decision. Solid arrows carry original bytes or the response chosen for the caller. Dotted arrows carry projected content or check decisions.
Invalid inspected representations are rejected before release. Reviewed successful responses without guarded text and upstream HTTP errors have operation-specific handling; follow buffered requests for those branches. Streaming releases approved windows in order and cannot retract earlier output.
Coverage and enabled checks are independent. This forwarding table makes the uninspected paths explicit:
Bindings currently use single_text.v1 and single_text_delta.v1 capabilities.
An operation name does not imply inspection of its entire upstream API. Consult
providers and streaming for the shipped content
restrictions and guard contracts for field-level coverage.
Design Decisions and Tradeoffs
These are current implementation choices and their observable consequences.
Refer to framework internals for the interfaces enforcing these boundaries and streaming inspection for a concrete window example.
Operating the Gateway
Construction, startup, request handling, and shutdown have different owners:
Timeouts do not establish that uncooperative owners released all resources. Embedding applications must explicitly enter the proxy application’s lifespan; mounting alone does not initialize it. Follow lifespan and embedding and retain the compatibility validation requirements.
Destination routing, replication, and placement are separate dimensions that can be combined:
For example, NeMo Guardrails Gateway can use an inference gateway as its fixed upstream and run as several equivalent replicas behind an ingress. Choosing an inference gateway does not determine replica count or require embedding.
Several provider mounts in one process share its HTTP client and transport and do not provide process isolation. The deployment guide supplies topologies and routing examples; configuration covers selection and limits without duplicating those deployment instructions here.