OpenAI | NeMo Gym

The openai_model server connects NeMo Gym to OpenAI without translating between API formats: a Responses request goes to OpenAI’s Responses API and returns a Responses object, and a Chat Completions request goes to OpenAI’s Chat Completions and returns a chat completion — OpenAI serves both natively.

Because it is a pass-through, the Responses endpoint also works with any other backend that implements the Responses API natively — point openai_base_url at it. In practice that is rare: most hosted backends only expose Chat Completions and instead go through Inference Providers, which translates between the two formats.

Gym pins OpenAI’s Python SDK to 2.45.0, including max reasoning effort and OpenAI’s programmatic tool calling schema. Responses requests keep typed prompt-cache options, the caller field of function and custom tool calls (whether the model issued the call directly or a program item issued it), and opaque program / program_output items through validation and continuation. Chat Completions and Anthropic conversion accept direct callers but reject program callers and program items because the link between a tool call and its program has no representation in those formats; drop_unrepresentable_request_fields does not drop that link. Both conversions reject non-null allowed_callers tool eligibility, including an empty list, and function output_schema definitions that they cannot preserve. These items do not acquire training metadata. Strict ATIF export rejects program items and non-null caller fields that ATIF cannot represent.

Unreported cache-write usage stays unknown internally; explicit zero and reported counts remain intact through usage aggregation. Synthesized Responses SSE omits an unknown cache_write_tokens instead of emitting null or inventing zero. This wire adaptation does not mutate captured usage or change provider-reported token totals.

For training workloads that require token IDs and log probabilities, use vLLM instead. Hosted endpoints do not expose the token-level information needed for RL training.

Supported APIs

This server exposes both endpoints and forwards each directly to the upstream endpoint:

OpenAI Responses — /v1/responses
OpenAI Chat Completions — /v1/chat/completions

Provider reasoning content

Responses-compatible providers sometimes use output_text for a reasoning item’s content. Before validating provider responses, this server changes that part’s discriminator to reasoning_text only when it contains a string text and no other fields except optional empty annotations: []. Text, encrypted content, item order and other response fields are preserved without modifying the provider’s response object. Unknown or malformed shapes still go through validation unchanged. Requests and continuation replay are unchanged.

Set Your Credentials

Store your values in env.yaml in the project root (gitignored):

policy_base_url: https://api.openai.com/v1
policy_api_key: your-api-key
policy_model_name: gpt-4o-mini

Point policy_base_url at any OpenAI-compatible Responses endpoint to reuse this server with a different host.

For NVCF endpoints, requests that take longer than 340 seconds to produce a first byte need the HTTP/2 PING sidecar.

Configuration Reference

Parameter	Type	Default	Description
`openai_base_url`	`str`	—	Required. Base URL of an endpoint that serves the Responses API.
`openai_api_key`	`str`	—	Required. API key for the endpoint.
`openai_model`	`str`	—	Required. Model identifier (for example, `gpt-4o-mini`).
`openai_default_headers`	`dict`	`{}`	Extra headers sent on every request.
`session_key_headers`	`list[str]`	`[]`	Names of headers that carry the current session key on every upstream request made for a rollout. Requires rollout correlation (see Keep a Session on One Replica). Requests outside a rollout get no generated header. Inside a rollout it replaces a header of the same name, in any case, from `openai_default_headers`; a header set on the request itself wins.
`extra_body`	`dict`	`{}`	Default parameters merged into every request body. Values set on the incoming request take precedence.
`max_concurrent_requests`	`int`	`null`	Cap on in-flight upstream requests (per-process, at least 1). `null` = unlimited; set it on rate-limited endpoints.
`max_http_attempts`	`int`	`3`	Bounded retries for transient HTTP failures (404, 408, 429, 500, 502, 503, 504, 520). Retries wait with exponential backoff and full jitter (1 s doubling, capped at 60 s), or for the answer’s `Retry-After` within the same cap. A 429 whose `error.code`/`error.type` is `budget_exceeded` or `insufficient_quota`, or a 401/403 whose body is an invalid key, trips the client immediately; later requests skip the wire. Generic “quota exceeded” 429s and other 401/403s still retry or return as before. A provider that refuses or drops connections gets the same number of connection attempts, 0.5 s apart, before the call fails with HTTP 500. Values of 1 or 2 also reduce request-timeout attempts from three to one or two; the default stays three.
`max_rate_limit_wait_seconds`	`float`	`3600`	Total wait for 429 and 503 answers, which are retried until it is spent instead of `max_http_attempts` times; the request fails once the next wait would pass it, even when `max_http_attempts` allows more tries. `0` keeps 429 and 503 on the `max_http_attempts` count.
`reasoning_effort_none_replacement`	`str`	`null`	Opt-in value reported in place of a response’s `reasoning.effort: "none"`, for callers that cannot accept `"none"`. `null` reports the provider’s value unchanged.
`upstream_max_num_tries`	`int`	`null`	Set to `1` so each provider call makes exactly one HTTP attempt. Disables the client’s transport and HTTP-status retries, replacing their generic-error and `max_http_attempts` limits. Required by an `upstream_retry_policy` with more than one attempt.
`upstream_request_timeout_seconds`	`float`	`1800`	Total time limit for each provider attempt, not a deadline for the whole retry schedule. `null` means no limit.
`upstream_connect_timeout_seconds`	`float`	`null`	Time limit for each attempt to obtain a connection, including waiting for a free aiohttp connection-pool slot. Requires `upstream_request_timeout_seconds`.
`upstream_pool_timeout_seconds`	`float`	`null`	Time limit for each attempt to wait for one of the `max_concurrent_requests` slots. Requires `max_concurrent_requests`.
`upstream_retry_policy`	`object`	one attempt	Model-server retry layer around provider calls: `max_attempts`, `pre_request_jitter_seconds`, `backoff_initial_seconds`, `backoff_multiplier`, `backoff_jitter_fraction`, and `terminal_http_status_codes`. Each attempt reacquires a concurrency slot. A spent or unauthorized key stops retries immediately; exhausted retries return HTTP 500.
`propagate_upstream_http_status_codes`	`list[int]`	`[]`	Upstream error statuses (400–599) returned to the caller with that status and the provider’s error body, instead of HTTP 500. A status that the retry policy retries until it is exhausted still returns HTTP 500.

The model is fixed by configuration. This server always sends the configured openai_model (from policy_model_name) to the upstream endpoint. If an incoming request carries its own model field — as standard OpenAI-compatible clients and SDKs do — that value is overwritten, so you cannot switch models on a per-request basis. To run a different model, change the config and start a new server.

Keep a Session on One Replica

A router that pins a session to one model replica reuses that replica’s KV cache only if every request of a session carries the same session key. Set session_key_headers to the header names your router reads, and turn on rollout correlation:

gym env start --config ${environment_config} --config ${model_config} \
    ++rollout_correlation=true \
    ++policy_model.responses_api_models.openai_model.session_key_headers='[X-Session-Id]'

rollout_correlation is a global option and defaults to false. It makes agents that use Gym’s agent helpers add the /ng-rollout/<id> prefix to their model calls, so the model server knows which rollout a call belongs to, without turning on model-call capture. observability_enabled and opted-in token capture add the prefix too.
session_key_headers defaults to [], so nothing changes unless you set it.
A request that carries no rollout identity gets no generated header. Static headers from openai_default_headers still apply to every request, inside or outside a rollout.
A session is a rollout attempt. The header carries the session key, which is the rollout’s capture key: <rollout id>, or <rollout id>-a<N> for a retry. Agents that make model calls inside the same attempt, such as a policy and a user simulator, or a judge that calls through this model server, share one session and therefore one replica.

Usage Example

1. Set model and environment config

environment_config="resources_servers/mcqa/configs/mcqa.yaml"
model_config="responses_api_models/openai_model/configs/openai_model.yaml"

2. Start servers

gym env start --config ${environment_config} --config ${model_config}

3. Evaluate your agent

gym eval run --no-serve --agent mcqa_simple_agent \
    --input resources_servers/mcqa/data/example.jsonl \
    --output results/mcqa_rollouts.jsonl