Skip to navigation

The HTTP transport sends requests from the gateway to the upstream model service. The default transport handles ordinary HTTP and HTTPS connections. Set base_url to choose an upstream; use a custom transport when you need different connection settings, custom TLS, or routing based on the requested model.

A transport changes how requests reach the upstream. Provider bindings still define the accepted payloads, and Rails still supplies the checks. This hook applies to the HTTP proxy’s outgoing provider requests; Rails models and checking services use their own clients.

Inject a Transport

Pass an http_transport_factory to the application factory. This is a function that creates the transport when the application starts. The gateway shares it across provider instances and closes it during shutdown.

Complete the Python setup, then replace proxy_app.py with the following example. It increases the shared connection limit to 200 for applications handling more concurrent requests:

import httpx
from nemoguardrails.server.experimental.application import create_proxy_application
from proxy_config import config
app = create_proxy_application(
config,
http_transport_factory=lambda: httpx.AsyncHTTPTransport(
limits=httpx.Limits(
max_connections=200,
max_keepalive_connections=40,
keepalive_expiry=5.0,
),
trust_env=False,
),
)

Choose limits for your workload and upstream capacity. These limits apply to the whole application; refer to connection-pool capacity for how concurrent streams share connections. The factory must create a new transport for each application lifespan.

For guarded operations, the transport is reached after input inspection and its returned content follows output inspection when enabled. It receives a complete outbound HTTP request, not the Rails resolver’s context object. If it handles several logical hosts or protocols, dispatch them explicitly.

Route Model Aliases

Use a model alias when clients should send a name such as public-model while your application chooses the provider model and destination. A routing transport looks up that name in a trusted table and supplies the destination’s credentials. Every destination must support the instance’s selected binding and dialect.

This Chat-only recipe uses the bundled model-routing example. Save it as routed_app.py beside proxy_config.py. It maps public-model to the same OpenAI model used in the quickstart:

from pathlib import Path
import httpx
from nemoguardrails.server.experimental.application import ConfigEntry, create_proxy_application
from nemoguardrails.server.experimental.examples.model_routing_transport import ModelRoute, ModelRoutingTransport
from nemoguardrails.server.experimental.providers.openai.integration import create_openai_chat_integration
routes = {
"deployment": ModelRoute(
base_url=httpx.URL("https://api.openai.com/v1"),
upstream_model="gpt-6-luna",
api_key_environment_variable="OPENAI_API_KEY",
),
}
def routing_transport():
return ModelRoutingTransport(routes, {"public-model": "deployment"}, default_model="deployment")
app = create_proxy_application(
providers=(create_openai_chat_integration(base_url="http://model-router.internal/v1"),),
rails_configs=(ConfigEntry("standard", Path("proxy-example/rails")),),
http_transport_factory=routing_transport,
)

Set OPENAI_API_KEY in the server environment through your deployment’s secret management, start uvicorn routed_app:app --host 127.0.0.1 --port 8000, and use model public-model in the Chat request. Unknown model names return a local error without contacting a provider. Requests without a model use the explicit default destination; their bodies do not acquire a model field.

The logical model-router.internal URL is a routing label handled by the transport, not a service to deploy. The example preserves non-JSON upload streams and encoded URL suffixes, replaces the logical base path, and limits JSON reads to 1 MiB by default (max_json_body_bytes). Oversized JSON receives HTTP 413 before network dispatch. The transport hook does not itself implement model balancing, retries, or another provider’s dialect.

The example removes caller Authorization, api-key, x-api-key, and Cookie headers and supplies the destination’s bearer credential. Review any additional credential headers used by your environment. Model translation changes request serialization and removes stale body signatures; this is an explicit exception to the default proxy forwarding, which preserves the original bytes. The response is still streamed through the selected binding and output checks. Every destination must match that binding; routing never switches dialects automatically.

Authorize callers and restrict exposed routes before supplying application-owned provider credentials. This recipe guards Chat creation only; unrelated routes retain passthrough and can reach the default destination without checks. Never take destination URLs or Rails paths from an untrusted request.

The runtime owns closing the returned transport. Your transport owns any network transport it creates and must implement cooperative, repeatable cleanup and the required streaming behavior. Refer to constraints and ownership.

Connection-Pool Capacity

The default pool is shared across provider instances within one application lifespan. Each server worker has its own runtime and pool. Without an explicit override, HTTPX’s defaults apply: 100 total connections, 20 idle keep-alive connections, and a five-second keep-alive expiry.

The default outbound transport uses HTTP/1.1. An active streaming response holds a connection until its body is consumed or closed. A busy provider can therefore use the available connections and make requests to another provider wait. Pool acquisition uses the application’s configured request timeout; exhaustion surfaces through the existing upstream-failure mapping.

Set connection limits on the transport as shown in the factory example. These are aggregate limits, not per-provider quotas. Size them for expected concurrent streams and upstream capacity. Raising the limit does not provide fairness or isolate providers from each other’s traffic.

Custom transports own their pool configuration. The factory creates a fresh transport for each application lifespan; the runtime owns and closes it.

Forwarding and Caller Isolation

ProviderTransport builds outbound requests before the HTTPX transport sees them. It removes hop-by-hop headers and the integration’s declared control headers. Provider authentication and API-version headers follow the forwarding rules; they are not automatically classified as proxy-only control headers.

The shared client pools connections, not caller sessions. It does not persist upstream cookies or apply client-level default authentication/headers to outgoing requests. Explicit caller Cookie and upstream Set-Cookie can pass through. Inject destination credentials in a trusted transport when needed, and remove credentials that must not cross origins.

Guarded requests and buffered responses use the configured body limits, including buffered responses with output checks disabled. Inspected streams also have event and pending-byte limits. Pass-through request and response bodies for unrelated operations stream without those guarded limits. For unchanged streamed requests, one valid Content-Length is retained; duplicate, malformed, or conflicting length headers are rejected. Bodyless HTTP/1 requests are sent without chunked framing. HTTP/2 and HTTP/3 requests without length headers continue streaming, since their bodies use protocol frames. Guarded request lengths are recalculated from the outgoing bytes. The pass-through router uses the instance’s shared ownership policy to reject local and mapped upstream aliases before opening an upstream request. Declared canonical pass-through methods remain available. Outer ASGI mounts are removed from decoded and raw paths together, preserving encoded suffixes and queries. All provider routers are validated before they are installed.

Register local routers through the factory before provider catch-alls. Nested router prefixes and dependencies are preserved, and unsupported methods on reserved local paths are rejected without upstream forwarding.

Only guarded operations with output inspection request identity encoding; pass-through traffic and uninspected guarded output preserve the caller’s Accept-Encoding value.

Manage Lifecycle and Failures

The factory is invoked once per application lifespan and must return a new httpx.AsyncBaseTransport. The runtime owns closing it through the shared HTTP client. Your transport owns child transports it creates. A factory result of the wrong type is rejected. If runtime construction fails after transport creation, the application attempts to close the transport while propagating the original error. Rails startup failure also triggers HTTP cleanup.

HTTP cleanup has a shielded five-second cooperative budget, separately from response cleanup, read timeouts, and the launcher’s request-drain deadline. Failures are logged without masking the primary error. Blocking or cancellation-suppressing custom code cannot be forcibly stopped by this budget. Expected network failures before response handoff use provider error mapping; cancellation and programming errors are not rewritten as network failures.

Constraints and Ownership

Follow these constraints when implementing a custom transport:

  • The transport is application-wide and sees requests for every configured provider.
  • A multi-provider transport must dispatch using logical hosts, paths, or other server-controlled metadata.
  • Provider catch-all routes stream request and response bodies without applying guarded-route buffer limits.
  • The factory must return a new transport for each lifespan; the application runtime closes it.
  • A production transport must preserve HTTPX streaming, cancellation, and response-close behavior.
  • Dispatch owns each response until handoff; the returned response then owns cleanup, including failure before iteration. Cooperative cleanup attempts are shielded and bounded, and failures are logged without replacing the primary outcome. Closing must tolerate retries; a timeout does not establish that resources were freed.
  • Response cleanup budgets are separate from transport read timeouts and application shutdown.
  • httpx.ASGITransport buffers ASGI responses and is not the production choice for guarded streaming.
  • Dynamic destinations must come from trusted configuration. Arbitrary client-provided URLs can enable server-side request forgery (SSRF).
  • Cross-origin routing must apply the credential belonging to the selected origin.
  • Keep Rails checks and provider field validation in the guarded proxy pipeline; the transport implements egress policy.