Skip to navigation

Start with the architecture overview for system boundaries and inspection coverage. This page follows guarded operations through execution. The buffered example enables input and output checks; the streaming examples enable output inspection with buffering before release.

Buffered Requests

The sequence is scoped to one buffered provider operation. The binding validates provider shapes and locates selected text. The runtime selects a preloaded checker and validates its policy. Request and response bodies retain their original bytes on accepted paths. Client and Upstream are external participants; the other lifelines are responsibilities inside the gateway process. Solid arrows are calls or requests; dashed arrows are returns. The alt frames show mutually exclusive paths, and opt marks conditional output checking.

opt [Guarded output text exists] alt [Output allowed or reviewed no-text result] [Output blocked, unsupported, or check fails] alt [Upstream HTTP error] [Successful response] alt [Input blocked or check fails] [Input allowed] Provider-native JSON request Validate request and locate input text Selected text and response mode Resolve registered Rails ID and checker policy Ready checker Check input text Input decision Provider-shaped error, no inference request Forward original request body Buffered provider response Relay upstream error without output inspection Validate response and locate output text Selected text or reviewed no-text result Check output with input context Output decision Original response body Provider-shaped error, withhold response body Client Guarded HTTP execution Provider binding Rails runtime Upstream

Invalid requests, resolver failures, and unsupported checker configurations also stop before dispatch. Successful inspected responses must satisfy their bound shape even when they contain no text. Current integrations reject replacement outcomes rather than rewriting provider content.

Input checks run only when configured; guarded request validation still applies when input checks are disabled. With output inspection disabled, the response bypasses output projections. HTTP limits and forwarding rules still apply to the selected path; refer to configuration.

For an executable example of byte preservation and input and output checking, see the Chat integration tests.

Caller-Visible Outcomes

For the shipped OpenAI bindings, the following outcomes describe the gateway’s error mapping. HTTP statuses below apply before downstream response headers are committed.

Some execution failures handled internally by Rails currently reach the gateway as BLOCKED results and are reported as content_blocked. A 400 response alone does not establish that a policy detected a violation. Inspect checking-service and application diagnostics before classifying an outage as a policy block. Structured failure propagation is tracked in #2485; its proposed behavior is not part of the current adapter.

OutcomeCaller SeesInference and Content Release
Input check blocksHTTP 400, content_blockedNo inference call.
Buffered output check blocksHTTP 400, content_blockedInference ran; its answer is withheld.
Check exception reaches the adapter, or checker returns an explicit failureHTTP 502, guardrails_input_check_failed or guardrails_output_check_failedInput failure prevents dispatch; output failure withholds the answer. Internally handled Rails failures can instead produce content_blocked, as noted above.
Checker requests replacementHTTP 422, request_content_not_replaceable, or HTTP 502, response_content_not_replaceableInput replacement prevents dispatch; output replacement withholds the answer.
Upstream returns an HTTP errorUpstream error response under forwarding rulesNo output inspection; this is an upstream outcome.
Inspected stream fails after headersProvider-shaped SSE error, if the connection is writablePending content is withheld; released content remains delivered. HTTP status cannot change.

The error mapping defines additional validation, routing, and limit failures. The Chat integration tests assert blocked and replacement outcomes. Provider-shaped errors describe a wire format; applications must still handle these gateway-specific codes. In particular, an HTTP success status alone does not establish that a stream completed.

Streaming Inspection

Streaming uses the same prepared input and input check. With output inspection enabled, the gateway parses server-sent event (SSE) boundaries, classifies provider events, and retains their original bytes while checking selected text windows. The streaming kernel validates text snapshots against observed deltas. Request-local hooks enforce provider ordering, completion, and error framing.

The example uses chunk_size: 2 and context_size: 1. Each numbered delta is a nonempty provider text delta. Sizes count deltas, not tokens, bytes, or all SSE events. Rectangles represent pending data or processing steps; the diamond is a check decision. Solid arrows show processing and release; the dotted arrow carries retained text context into the next check.

First checked window Next checked window Receive next window Previously approved delta 2 Cannot retract Project text Allow Project new text Check result Allow Block Events 1 and 2 remain delivered Pending original events: deltas 1 and 2 Check text from deltas 1 and 2 Allowed: release original events 1 and 2 Pending original events: deltas 3 and 4 Check delta 2 as context plusdeltas 3 and 4 Decision Release original events 3 and 4 Withhold pending events and emitprovider-shaped error event

Context participates in a later check without being sent twice. Pending metadata and snapshots preserve their order relative to guarded text. Validated data-less keepalives can be released promptly when no payload is pending. The final partial window is checked at completion, so short responses can remain buffered until the end.

Output inspection requires streaming checks to be enabled and stream_first: false. Provider hooks enforce declared terminal behavior: Chat uses its reviewed completion handling and [DONE]; Responses requires a native terminal event. A truncated event fails without releasing its unfinished bytes. Refer to providers and streaming for binding-specific coverage and settings, and the streaming tests for release ordering and withheld content.

Streaming Errors and Cleanup

This state view is scoped to an output-inspected stream. Arrows are lifecycle transitions, not data dependencies. Rounded boxes are states, and the filled circles mark entry and exit. The two cleanup states represent the same ownership obligation on different response paths. The response-header transition determines whether a subsequent failure can change the HTTP status.

Before returning a pre-header HTTP error, the gateway attempts cleanup of any acquired upstream owner. Accepting upstream response headers does not validate SSE events: event parsing and protocol checks run during body iteration, after downstream headers are sent. A malformed first event therefore produces a streaming error after header commitment. Processing failures in the state view include check failures, protocol drift, and event-size or pending-byte limit violations. Normal completion requires valid terminal behavior and approval of any final partial window.

An error event can reach the client only while the downstream connection remains writable. Already released content remains delivered after a failure. A reviewed upstream HTTP error is relayed before streaming begins, and reviewed provider error events follow the binding’s provider-aware handling.

The application response owner closes the upstream source independently of the body iterator, including failure before iteration starts. Iterated guarded streams also attempt cleanup in their finalization path. Cleanup is cooperative, bounded, shielded, and must tolerate repeat attempts; finishing response handling does not prove that an uncooperative owner released every resource.

Event-size and pending-byte limits reject oversized inspected streams. They are not a complete process memory budget and do not impose universal bounds on unrelated pass-through traffic. Read/idle timeouts and cleanup budgets are separate controls.

Follow streaming HTTP ownership, the streaming kernel, and their HTTP tests and kernel tests. For application-wide owners, refer to framework state ownership.