nemo_gym.models.base_responses_api_model
nemo_gym.models.base_responses_api_model
Model server base classes.
Every Gym model server derives from SimpleResponsesAPIModel.
It serves the three model dialects: /v1/responses, /v1/chat/completions, and /v1/messages.
It also installs the model-call capture middleware from nemo_gym.models.model_call_capture.
Module Contents
Classes
Functions
Data
PROPAGATED_REFUSAL_CODE_ATTRIBUTE
API
Bases: BaseServer
Bases: BaseRunServerInstanceConfig
Bases: BaseModel
A model server instance’s checkpoint block.
Bases: BaseResponsesAPIModel, SimpleServer
Finalize capture after conversion to the response returned to the client.
Whether the ids of the responses responses() returns are this server’s own.
A keepalive stream (stream_keepalive_s) names its response before responses() returns,
so that response takes the announced id. An id an upstream Responses API issued must reach the
client unchanged, so a server that serves those cannot keep a stream alive.
Keep an opened Responses stream alive until pending, which serves response_id, completes.
The headers went out before pending finished, so the terminal event carries the request’s failure report.
Apply this server’s tool policy while mapping a streaming body to the strict shape.
Every /v1/responses dispatch that accepts the streaming wire dialect sanitizes
through here, so the config’s drop_custom_tools, drop_hosted_tools and
custom_tool_strategy settings reach the sanitizer on each of them.
Allow adapters to retain an explicit upstream-status compatibility policy.
Stream a completed response; until it completes nothing is sent to the client.
Serve the same converted response to capture and Responses SSE clients.
Serialize buffered SSE before committing externally staged capture.
The (code, message) of the response.failed event that reports a failure after keepalives.
Mirrors what the client would have seen before the stream opened: a refusal the server
propagates keeps its OpenAI error code (context_length_exceeded tells a harness to compact
and retry), and an upstream HTTP error follows the server’s propagation policy. Upstream
errors never report their request URL.
Validate server-specific request options before recording capture intent.
Default /v1/chat/completions entrypoint shared by every Gym model server.
A non-streaming request validates strictly against
NeMoGymChatCompletionCreateParamsNonStreaming and delegates to this server’s own
chat_completions(), preserving the historical non-streaming behavior (including the
standard 422 shape). When the client sends stream: true (blackbox
Chat-Completions-over-SSE harnesses like the OpenClaw agent always do), the request is
sanitized onto that same strict model (drop stream/stream_options; see
nemo_gym.models.chat_streaming), validated identically, delegated to the same
chat_completions(), and the complete response is buffered and re-emitted as a
synthesized chat.completion.chunk SSE stream. Keepalive comments maintain the
connection while the backend computes; model output is still buffer-then-replay.
Only a genuine boolean stream: true takes the streaming path; any other value
(e.g. "false" or 1) stays on the strict non-streaming path, which rejects the
malformed stream with the same 422 as before.
OpenAI-compatible POST /v1/embeddings default: 501 unless a server overrides it.
Servers backed by an embeddings-capable provider (e.g. openai_model,
inference_provider) override this to pass body through to the provider and
return the provider’s JSON unchanged.
The URL root where the inference worker serving base_url takes generation cuts.
Default Anthropic Messages <-> Responses mapping shared by every Gym model server.
Translates the inbound Anthropic Messages request to the Responses API, delegates to this
server’s own responses() (so it reuses whatever backend the server has), and maps the
result back to an Anthropic Messages response. When the client requested stream: true
(the Claude Code CLI always does), the complete response is re-emitted as a synthesized
Anthropic SSE event stream. Servers may override this for native Messages handling.
Default /v1/responses entrypoint shared by every Gym model server.
A plain JSON request validates strictly against
NeMoGymResponseCreateParamsNonStreaming and delegates to this server’s own
responses(), preserving the historical non-streaming behavior. When the client
requests stream: true (blackbox Responses-over-SSE harnesses like the Codex CLI
always do), the request is first sanitized from the streaming wire dialect (extra
bookkeeping fields, namespace tool specs — see nemo_gym.models.responses_streaming),
delegated to the same responses(), and the complete response is re-emitted as a
synthesized Responses SSE event stream. Upstream HTTP errors and HTTPException
retain their status before the stream starts. Other responses() failures become terminal
response.failed events; bad-request validation still fails eagerly. With
stream_keepalive_s set, a call that outlasts it opens the stream early and keeps it alive
with response.in_progress events, and any later failure becomes response.failed.
Serialize a completed non-streaming model response with orjson.
FastAPI serializes a bare dictionary or Pydantic model with jsonable_encoder.
That function recursively visits every token ID and log probability before standard-library json.dumps runs.
Convert Pydantic models to JSON-compatible values before encoding the result.
Return the encoded bytes as a Response so FastAPI does not encode them again.
Preserve Response objects created by model-server overrides.
Reduce a request field to plain data so equal schemas serialize equally.
Handlers can expose tools as dictionaries or Pydantic models. Normalization keeps an unchanged schema stable across handlers.
Return prompt-shaping request fields that are not turns.
Instructions and tools can be siblings of the message list. The chat template renders both into the prompt. Including them prevents prefix reuse across different request envelopes.
Return the conversation carried by any supported dialect.
Lineage uses model-authored turns to identify the parent call.
Chat and Anthropic use messages.
Responses uses input.
The request envelope is prepended as a pseudo-turn because it also shapes the prompt.
Validate a /v1/chat/completions body dict, surfacing failures as FastAPI’s standard 422.
include_url=False mirrors FastAPI’s native body validation, which strips the
errors.pydantic.dev url from each detail entry, so the 422 body stays byte-for-byte
identical to the previous typed-Body() binding.
Validate a /v1/responses body dict, surfacing failures as FastAPI’s standard 422.
Return (message, code, param) for a marked engine refusal.
The engine’s error body travels on the exception as response_content (attached by
nemo_gym.client.http_client.raise_for_status). vLLM 0.10.1 and later format it as
{"error": {"message": ..., "type": ..., "param": ..., "code": 400}}; older engines used a
flat object with a top-level message. The message is the engine’s own sentence from
either shape, verbatim; any other body is returned as raw text so no information is lost.
The param is the engine’s param when its body names one. The code is whatever the server
stored under PROPAGATED_REFUSAL_CODE_ATTRIBUTE when it marked the error.
Whether a model server marked error as a refusal it propagates on purpose.