> For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/nemo/guardrails/latest/run-guardrailed-inference/using-gateway/llms.txt. # NeMo Guardrails Gateway > Apply NeMo Guardrails to existing model APIs with configurable checks that preserve supported provider formats and work with your inference infrastructure. NeMo Guardrails Gateway provides guardrails enforcement for native model APIs. Apply the NVIDIA NeMo Guardrails library's configurable checks between your applications and model services while preserving supported provider formats and keeping your existing inference infrastructure. The gateway runs as an HTTP reverse proxy. Provider bindings select the content to inspect, and Rails supplies the checks. The proxy layer forwards accepted provider JSON and server-sent event (SSE) bytes without rebuilding them from the validation models. Your application or surrounding infrastructure owns access controls, model routing, budgets, and retry policies. > **Note** > > The gateway currently supports a limited single-text inspection profile. Review > [provider coverage](/run-guardrailed-inference/using-gateway/providers-and-streaming#coverage-of-the-shipped-profiles) > before connecting your application. > > It uses `python -m nemoguardrails.server.experimental`. The [Guardrails API server](/run-guardrailed-inference/using-fastapi-server) (`nemoguardrails server`) > and [hooks server](/run-guardrailed-inference/using-hooks-server) are separate > applications with different commands and configuration contracts. ## Start Here [Run the gateway](/run-guardrailed-inference/using-gateway/quickstart), send a request, and verify that input is blocked before inference using the bundled example. Then choose your next task: #### [Quickstart](/run-guardrailed-inference/using-gateway/quickstart) Start with OpenAI, then call the gateway from a shell or Python. #### [Guard Contracts](/run-guardrailed-inference/using-gateway/guard-contracts) Review which content is guarded, constrained, or opaque. #### [Configuration](/run-guardrailed-inference/using-gateway/configuration) Select upstreams, operation bindings, and Rails configurations. #### [Python Integration](/run-guardrailed-inference/using-gateway/python-integration) Construct an application or embed it in an ASGI host. #### [Deployment](/run-guardrailed-inference/using-gateway/deployment) Plan replicas, access controls, readiness, and shutdown. #### [Troubleshooting](/run-guardrailed-inference/using-gateway/troubleshooting) Diagnose startup, request, streaming, and cleanup failures. Check [providers and streaming](/run-guardrailed-inference/using-gateway/providers-and-streaming) for accepted shapes and window settings, or read the [architecture overview](/run-guardrailed-inference/using-gateway/architecture). The architecture section also covers the [request lifecycle](/run-guardrailed-inference/using-gateway/architecture/request-lifecycle) and [framework internals](/run-guardrailed-inference/using-gateway/architecture/framework-internals). ## Advanced Configuration Use these guides for detailed customization, resource ownership, and boundary review: | Task | Guide | | ------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------ | | Select Rails from trusted request context | [Rails configuration resolution](/run-guardrailed-inference/using-gateway/advanced/configuration-resolution) | | Customize outbound HTTP or route model aliases | [HTTP transport customization](/run-guardrailed-inference/using-gateway/advanced/http-transport) | | Configure several mounts, upstreams, and policies | [Multiple provider instances](/run-guardrailed-inference/using-gateway/advanced/multiple-provider-instances) | | Customize request values and response metadata, then export the contract | [Request and response policies](/run-guardrailed-inference/using-gateway/advanced/extend-provider-bindings) | ## How the Pieces Fit The following components define routing, inspection, and resource ownership: | Piece | Responsibility | | --------------------------------------- | --------------------------------------------------------------------------------- | | Application configuration | Choose upstream instances, operation bindings, dialects, and Rails IDs. | | Provider binding and capability profile | Describe which content the current integration and pipeline can handle. | | Rails configuration | Choose checks for that content and their dependencies. | | Guard contract | Make the bound inspection coverage reviewable as an export. | | HTTP transport | Perform the outbound exchange, with optional application-specific network policy. | Check [provider coverage](/run-guardrailed-inference/using-gateway/providers-and-streaming#coverage-of-the-shipped-profiles) before connecting your application. The built-in bindings currently support a single user text input and one assistant text target; additional content needs matching binding and checking support. Start with the [quickstart](/run-guardrailed-inference/using-gateway/quickstart) for the library's `server` extra and installed examples. For exact declaration details, follow the technical references linked from each guide. ## Docs - [Run NeMo Guardrails Gateway](https://docs.nvidia.com/nemo/guardrails/latest/run-guardrailed-inference/using-gateway/quickstart.md): Start NeMo Guardrails Gateway for OpenAI, send buffered and streaming requests from a shell or Python, and block input before inference. - [Configure the Gateway](https://docs.nvidia.com/nemo/guardrails/latest/run-guardrailed-inference/using-gateway/configuration.md): Choose provider instances, operations, dialects, and Rails configurations without coupling routing to inspection policy. - [Providers, Profiles, and Streaming](https://docs.nvidia.com/nemo/guardrails/latest/run-guardrailed-inference/using-gateway/providers-and-streaming.md): Check supported request shapes, select a provider dialect, and tune streamed output checks. - [Understand Guard Contracts](https://docs.nvidia.com/nemo/guardrails/latest/run-guardrailed-inference/using-gateway/guard-contracts.md): Review what a provider binding inspects, constrains, and passes through using contracts exported from its Python declarations. - [Compose and Extend in Python](https://docs.nvidia.com/nemo/guardrails/latest/run-guardrailed-inference/using-gateway/python-integration.md): Build the gateway's HTTP proxy application, select Rails from trusted context, and extend HTTP dispatch or provider bindings. - [Deploy and Route Requests](https://docs.nvidia.com/nemo/guardrails/latest/run-guardrailed-inference/using-gateway/deployment.md): Choose a topology for one upstream, multiple provider instances, replicas, gateways, or embedded applications. - [Troubleshoot NeMo Guardrails Gateway](https://docs.nvidia.com/nemo/guardrails/latest/run-guardrailed-inference/using-gateway/troubleshooting.md): Diagnose startup failures, rejected payloads, checker errors, delayed streams, disconnects, and incomplete cleanup. - [Rails Configuration Resolution](https://docs.nvidia.com/nemo/guardrails/latest/run-guardrailed-inference/using-gateway/advanced/configuration-resolution.md): Select preloaded Rails configurations from trusted request context and manage resolver deadlines and lifecycle. - [HTTP Transport Customization](https://docs.nvidia.com/nemo/guardrails/latest/run-guardrailed-inference/using-gateway/advanced/http-transport.md): Inject outbound HTTP behavior, route model aliases, tune shared pools, and preserve streaming ownership. - [Multiple Provider Instances and Routing](https://docs.nvidia.com/nemo/guardrails/latest/run-guardrailed-inference/using-gateway/advanced/multiple-provider-instances.md): Compose mounts, policies and upstreams while understanding shared resources, route ownership and isolation. - [Customize Request and Response Policies](https://docs.nvidia.com/nemo/guardrails/latest/run-guardrailed-inference/using-gateway/advanced/extend-provider-bindings.md): Customize accepted request values and response metadata for buffered and streaming traffic, then export the bound guard contract. - [Architecture](https://docs.nvidia.com/nemo/guardrails/latest/run-guardrailed-inference/using-gateway/architecture.md): Understand NeMo Guardrails Gateway's boundaries, inspection model, design tradeoffs, and deployment responsibilities. - [Request Lifecycle](https://docs.nvidia.com/nemo/guardrails/latest/run-guardrailed-inference/using-gateway/architecture/request-lifecycle.md): Follow buffered and streamed requests through validation, checking, byte release, errors, and cleanup. - [Framework Internals](https://docs.nvidia.com/nemo/guardrails/latest/run-guardrailed-inference/using-gateway/architecture/framework-internals.md): Understand provider-neutral execution, composition hooks, trusted policy selection, and ownership when extending the gateway.