> This page is for version v0.25.0.
> For other versions, use one of these documentation indexes:
> - Latest (default): https://docs.nvidia.com/nemo/guardrails/latest/llms.txt
> - v0.25.0: https://docs.nvidia.com/nemo/guardrails/v0.25.0/llms.txt
> - v0.24.1: https://docs.nvidia.com/nemo/guardrails/v0.24.1/llms.txt
> - v0.24.0: https://docs.nvidia.com/nemo/guardrails/v0.24.0/llms.txt
> - v0.23.0: https://docs.nvidia.com/nemo/guardrails/v0.23.0/llms.txt
> - v0.22.0: https://docs.nvidia.com/nemo/guardrails/v0.22.0/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/guardrails/llms.txt.

# HTTP Transport Customization

> Inject outbound HTTP behavior, route model aliases, tune shared pools, and preserve streaming ownership.

The HTTP transport sends requests from the gateway to the upstream model service.
The default transport handles ordinary HTTP and HTTPS connections. Set
`base_url` to choose an upstream; use a custom transport when you need different
connection settings, custom TLS, or routing based on the requested model.

A transport changes how requests reach the upstream. Provider bindings still
define the accepted payloads, and Rails still supplies the checks. This hook
applies to the HTTP proxy's outgoing provider requests; Rails models and checking
services use their own clients.

## Inject a Transport

Pass an `http_transport_factory` to the application factory. This is a function
that creates the transport when the application starts. The gateway shares it
across provider instances and closes it during shutdown.

Complete the [Python setup](/nemo/guardrails/v0.25.0/run-guardrailed-inference/using-gateway/python-integration#create-an-application),
then replace `proxy_app.py` with the following example. It increases the shared
connection limit to 200 for applications handling more concurrent requests:

```python
import httpx

from nemoguardrails.server.experimental.application import create_proxy_application

from proxy_config import config

app = create_proxy_application(
    config,
    http_transport_factory=lambda: httpx.AsyncHTTPTransport(
        limits=httpx.Limits(
            max_connections=200,
            max_keepalive_connections=40,
            keepalive_expiry=5.0,
        ),
        trust_env=False,
    ),
)
```

Choose limits for your workload and upstream capacity. These limits apply to
the whole application; refer to [connection-pool capacity](#connection-pool-capacity)
for how concurrent streams share connections. The factory must create a new
transport for each application lifespan.

For guarded operations, the transport is reached after input inspection and
its returned content follows output inspection when enabled. It receives a
complete outbound HTTP request, not the Rails resolver's context object.
If it handles several logical hosts or protocols, dispatch them explicitly.

## Route Model Aliases

Use a model alias when clients should send a name such as `public-model` while
your application chooses the provider model and destination. A routing transport
looks up that name in a trusted table and supplies the destination's credentials.
Every destination must support the instance's selected binding and dialect.

This Chat-only recipe uses the bundled
[model-routing example](https://github.com/NVIDIA-NeMo/Guardrails/blob/develop/nemoguardrails/server/experimental/examples/model_routing_transport.py).
Save it as `routed_app.py` beside `proxy_config.py`. It maps `public-model` to the same
OpenAI model used in the quickstart:

```python
from pathlib import Path

import httpx

from nemoguardrails.server.experimental.application import ConfigEntry, create_proxy_application
from nemoguardrails.server.experimental.examples.model_routing_transport import ModelRoute, ModelRoutingTransport
from nemoguardrails.server.experimental.providers.openai.integration import create_openai_chat_integration

routes = {
    "deployment": ModelRoute(
        base_url=httpx.URL("https://api.openai.com/v1"),
        upstream_model="gpt-6-luna",
        api_key_environment_variable="OPENAI_API_KEY",
    ),
}

def routing_transport():
    return ModelRoutingTransport(routes, {"public-model": "deployment"}, default_model="deployment")

app = create_proxy_application(
    providers=(create_openai_chat_integration(base_url="http://model-router.internal/v1"),),
    rails_configs=(ConfigEntry("standard", Path("proxy-example/rails")),),
    http_transport_factory=routing_transport,
)
```

Set `OPENAI_API_KEY` in the server environment through your deployment's
secret management, start
`uvicorn routed_app:app --host 127.0.0.1 --port 8000`, and use model
`public-model` in the Chat request. Unknown model names return a local error
without contacting a provider. Requests without a model use the explicit default
destination; their bodies do not acquire a model field.

The logical `model-router.internal` URL is a routing label handled by the
transport, not a service to deploy. The example preserves non-JSON upload
streams and encoded URL suffixes, replaces the logical base path, and limits
JSON reads to 1 MiB by default (`max_json_body_bytes`). Oversized JSON receives
HTTP 413 before network dispatch. The transport hook does not itself implement
model balancing, retries, or another provider's dialect.

The example removes caller `Authorization`, `api-key`, `x-api-key`, and `Cookie`
headers and supplies the destination's bearer credential. Review any additional
credential headers used by your environment. Model translation changes request
serialization and removes stale body signatures; this is an explicit exception
to the default proxy forwarding, which preserves the original bytes. The response is
still streamed through the selected binding and output checks. Every destination must match
that binding; routing never switches dialects automatically.

> **Warning**
>
> Authorize callers and restrict exposed routes before supplying application-owned
> provider credentials. This recipe guards Chat creation only; unrelated routes
> retain passthrough and can reach the default destination without checks. Never
> take destination URLs or Rails paths from an untrusted request.

The runtime owns closing the returned transport. Your transport owns any network
transport it creates and must implement cooperative, repeatable cleanup and the
required streaming behavior. Refer to [constraints and ownership](#constraints-and-ownership).

## Connection-Pool Capacity

The default pool is shared across provider instances within one application
lifespan. Each server worker has its own runtime and pool. Without an explicit
override, HTTPX's defaults apply: 100 total connections, 20 idle keep-alive
connections, and a five-second keep-alive expiry.

The default outbound transport uses HTTP/1.1. An active streaming response holds
a connection until its body is consumed or closed. A busy provider can therefore
use the available connections and make requests to another provider wait. Pool
acquisition uses the application's configured request `timeout`; exhaustion
surfaces through the existing upstream-failure mapping.

Set connection limits on the transport as shown in
[the factory example](#inject-a-transport). These are aggregate limits, not
per-provider quotas. Size them for expected concurrent streams and upstream
capacity. Raising the limit does not
provide fairness or isolate providers from each other's traffic.

Custom transports own their pool configuration. The factory creates a fresh
transport for each application lifespan; the runtime owns and closes it.

## Forwarding and Caller Isolation

`ProviderTransport` builds outbound requests before the HTTPX transport sees
them. It removes hop-by-hop headers and the integration's declared control
headers. Provider authentication and API-version headers follow the forwarding
rules; they are not automatically classified as proxy-only control headers.

The shared client pools connections, not caller sessions. It does not persist
upstream cookies or apply client-level default authentication/headers to outgoing
requests. Explicit caller `Cookie` and upstream `Set-Cookie` can pass through.
Inject destination credentials in a trusted transport when needed, and remove
credentials that must not cross origins.

Guarded requests and buffered responses use the configured body limits,
including buffered responses with output checks disabled. Inspected streams
also have event and pending-byte limits. Pass-through request and response
bodies for unrelated operations stream without those guarded limits.
For unchanged streamed requests, one valid `Content-Length` is retained;
duplicate, malformed, or conflicting length headers are rejected. Bodyless
HTTP/1 requests are sent without chunked framing. HTTP/2 and HTTP/3 requests
without length headers continue streaming, since their bodies use protocol
frames. Guarded request lengths are recalculated from the outgoing bytes.
The pass-through router uses the instance's shared ownership policy to reject
local and mapped upstream aliases before opening an upstream request. Declared
canonical pass-through methods remain available. Outer ASGI mounts are removed
from decoded and raw paths together, preserving encoded suffixes and queries.
All provider routers are validated before they are installed.

Register [local routers](/nemo/guardrails/v0.25.0/run-guardrailed-inference/using-gateway/python-integration#local-application-routes)
through the factory before provider catch-alls. Nested router prefixes and
dependencies are preserved, and unsupported methods on reserved local paths
are rejected without upstream forwarding.

Only guarded operations with output inspection request identity encoding;
pass-through traffic and uninspected guarded output preserve the caller's
`Accept-Encoding` value.

## Manage Lifecycle and Failures

The factory is invoked once per application lifespan and must return a new
`httpx.AsyncBaseTransport`. The runtime owns closing it through the shared HTTP
client. Your transport owns child transports it creates. A factory result of
the wrong type is rejected. If runtime construction fails after transport
creation, the application attempts to close the transport while propagating
the original error. Rails startup failure also triggers HTTP cleanup.

HTTP cleanup has a shielded five-second cooperative budget, separately from
response cleanup, read timeouts, and the launcher's request-drain deadline.
Failures are logged without masking the primary error. Blocking or
cancellation-suppressing custom code cannot be forcibly stopped by this budget.
Expected network failures before response handoff use provider error mapping;
cancellation and programming errors are not rewritten as network failures.

## Constraints and Ownership

Follow these constraints when implementing a custom transport:

* The transport is application-wide and sees requests for every configured provider.
* A multi-provider transport must dispatch using logical hosts, paths, or other server-controlled metadata.
* Provider catch-all routes stream request and response bodies without applying guarded-route buffer limits.
* The factory must return a new transport for each lifespan; the application runtime closes it.
* A production transport must preserve HTTPX streaming, cancellation, and response-close behavior.
* Dispatch owns each response until handoff; the returned response then owns cleanup, including failure before iteration. Cooperative cleanup attempts are shielded and bounded, and failures are logged without replacing the primary outcome. Closing must tolerate retries; a timeout does not establish that resources were freed.
* Response cleanup budgets are separate from transport read timeouts and application shutdown.
* `httpx.ASGITransport` buffers ASGI responses and is not the production choice for guarded streaming.
* Dynamic destinations must come from trusted configuration. Arbitrary client-provided URLs can enable server-side request forgery (SSRF).
* Cross-origin routing must apply the credential belonging to the selected origin.
* Keep Rails checks and provider field validation in the guarded proxy pipeline; the transport implements egress policy.