Skip to navigation

NeMo Guardrails Gateway adds content inspection around an existing provider HTTP exchange. Provider bindings define the inspection coverage, the NVIDIA NeMo Guardrails library runs configured checks, and the upstream service generates the answer. Its HTTP proxy layer preserves accepted provider JSON bodies and server-sent events (SSE) while applying its path and header forwarding rules.

What Changes in Your Application

Point your provider client’s base URL at the chosen gateway mount, including its API prefix; the quickstart shows the request URL. Continue to send the selected provider operation’s JSON shape. Check the shipped coverage first: a working upstream request can still be outside the gateway’s guarded profile, such as a Chat request containing conversation history or tools.

For a supported Chat request, the binding selects the user message string for input checks. It constrains the message role and shape, while metadata such as model remains opaque. Output checks receive the selected assistant text and that user input as context. The original JSON bytes travel to the provider or caller only when the relevant validation and checks allow them.

Handle gateway errors as well as upstream errors. A blocked input prevents the inference call; a blocked buffered output withholds the answer. During streaming, a later error can follow already delivered text, so consume the stream’s error format as well as its HTTP status. Refer to caller-visible outcomes.

Choose a Reading Path

Choose the guide that matches your role:

You AreRead
Using the gateway in an applicationStart with system context and inspection and forwarding, then follow the request lifecycle.
Extending the frameworkRead framework internals for dependency boundaries, policy selection, state ownership, and implementation/test locations.
Operating or embedding a deploymentRead operating the gateway, then use the configuration, deployment, and Python integration guides.

System Context

This view shows the gateway as one software system and its external interactions. Rectangles represent systems or application-owned responsibilities. Solid arrows describe provider traffic; dotted arrows describe optional checking-service calls. Access controls belong to your application or ingress gateway.

Provider API request Authorized request Inferencerequest Inferenceresponse Provider response or guardrail error Return response Selectedcontent Checkresult Application or provider API client Host application or ingressgateway: inbound access controls NeMo Guardrails Gateway:provider HTTP forwarding andconfigured inspection Inference service or modelgateway: answer generation Optional configured checking services

The access-controls box represents a responsibility, not a required extra process. It can run in an embedding application’s middleware or an inbound gateway. The launcher binds loopback by default and does not authenticate callers. Checking-service calls serve a separate role from answer generation; the same backend can serve both roles. External services are needed only when your Rails checks use them.

The gateway is included in the NVIDIA NeMo Guardrails library under nemoguardrails.server.experimental and requires the library’s server extra. The library’s existing API server remains a separate application. YAML and Python entry points use the same proxy application factory and lifespan; refer to application assembly.

Inspection and Forwarding

Inspection uses a selected view of the payload. The gateway retains the original body or event bytes alongside that view and uses check decisions to control their release. It does not serialize an accepted response from a validation model.

This diagram shows a successful upstream response containing guarded text, with output inspection enabled. Rectangles are processing steps or retained data; the diamond is a release decision. Solid arrows carry original bytes or the response chosen for the caller. Dotted arrows carry projected content or check decisions.

Original bytes Read without rebuilding Selected text and input context Allow, block, or failure Retained bytes await decision Allow: release original bytes Block or fail HTTP error or SSE error event Successful provider response Retained original JSON body orpending SSE events Binding validates shape and selects text Configured output checks Release decision Withhold pending content andrender provider-shaped error Caller

Invalid inspected representations are rejected before release. Reviewed successful responses without guarded text and upstream HTTP errors have operation-specific handling; follow buffered requests for those branches. Streaming releases approved windows in order and cannot retract earlier output.

Coverage and enabled checks are independent. This forwarding table makes the uninspected paths explicit:

TrafficRequest HandlingResponse Handling
Guarded operation with checks enabledValidate the bound shape; run enabled input checks before dispatch.Apply enabled output checks to declared content; release accepted original bytes.
Guarded operation with output checks disabledGuarded request validation and any enabled input checks still apply.Relay response bytes without output projections or SSE classification.
Unselected operation handled by pass-throughForward under the provider’s HTTP routing and header rules, without content inspection.Forward without content inspection.
Unsupported content on a guarded operationReject unsupported request content before dispatch.With output inspection enabled, withhold unsupported successful response content. No fallback to pass-through.

Bindings currently use single_text.v1 and single_text_delta.v1 capabilities. An operation name does not imply inspection of its entire upstream API. Consult providers and streaming for the shipped content restrictions and guard contracts for field-level coverage.

Design Decisions and Tradeoffs

These are current implementation choices and their observable consequences.

ChoiceEffectTradeoff
Inspect projections while retaining original bytes.Accepted provider bodies/events keep their representation and metadata.Current integrations reject replacement outcomes; opaque fields remain uninspected.
Separate provider bindings, execution, and Rails checks.The same checking pipeline serves reviewed operations and dialects.New content capabilities need matching projection, execution, checker, and test support. A new profile name alone adds no capability.
Derive guard-contract exports from typed declarations.Review artifacts describe the bound coverage from one authoritative source.Editing exported YAML changes no runtime policy; stateful hooks and arbitrary Python behavior need separate review.
Preload every registered Rails configuration before readiness.Requests select ready checkers without loading files or constructing engines.Startup time and resources grow with the catalog, including unused entries. Initialization failure prevents readiness.
Check streaming windows before release.Output can arrive incrementally after checks approve each window.Smaller windows increase check frequency; short responses can wait until completion. Finite context and irreversible release limit whole-answer guarantees.

Refer to framework internals for the interfaces enforcing these boundaries and streaming inspection for a concrete window example.

Operating the Gateway

Construction, startup, request handling, and shutdown have different owners:

PhaseOwnerWhat Happens
Application constructionFactory, invoked by the launcher or Python hostValidate declarations, configuration paths, and route composition. No upstream calls or Rails initialization.
Lifespan startupRuntime and Rails repositoryCreate the shared HTTP client and transport and initialize every declared Rails configuration before readiness.
Request handlingGuarded route and response ownerSelect a ready checker; create request-local stream hooks/adapters and acquire upstream response resources as needed.
Failed startup or shutdownRuntime and Rails repository; host drives the lifespanAttempt checker cleanup and HTTP cleanup independently with bounded, shielded cooperative budgets.

Timeouts do not establish that uncooperative owners released all resources. Embedding applications must explicitly enter the proxy application’s lifespan; mounting alone does not initialize it. Follow lifespan and embedding and retain the compatibility validation requirements.

Destination routing, replication, and placement are separate dimensions that can be combined:

DecisionChoices and Consequences
Destination routingUse a fixed upstream, an inference gateway, distinct provider mounts, or a trusted application transport. The selected upstream must match the bound operation and dialect.
ReplicationEquivalent replicas each own independent client pools and Rails engines. An established stream stays with its handling replica. Replicas can front either a fixed upstream or an inference gateway.
PlacementRun a standalone service, local companion, or embedded ASGI application. Embedding transfers lifespan coordination to the host application.
Access and credentialsYour application or ingress gateway owns inbound authentication and authorization. The default transport forwards caller upstream credentials under its header rules. Destination-aware routing must remove inappropriate caller credentials and supply credentials for the selected destination.

For example, NeMo Guardrails Gateway can use an inference gateway as its fixed upstream and run as several equivalent replicas behind an ingress. Choosing an inference gateway does not determine replica count or require embedding.

Several provider mounts in one process share its HTTP client and transport and do not provide process isolation. The deployment guide supplies topologies and routing examples; configuration covers selection and limits without duplicating those deployment instructions here.