nemo_gym.server_runtime.base_environment_server
nemo_gym.server_runtime.base_environment_server
Shared lifecycle for environment servers.
Module Contents
Classes
Functions
Data
_RETRYABLE_CLIENT_ERROR_STATUSES
API
Bases: SimpleServer, Generic[EpisodeRequestT, EpisodeResponseT]
Expose a typed episode protocol with shared limits and cleanup.
Aggregate per-rollout scores into task-level metrics.
Record that a protocol step completed; state names the next step and what it needs.
While a checkpoint is open the episode parks here until resume.
A replacement attempt
that continues a checkpoint receives the latest recorded state from checkpoint_continuation.
Pass a function when building state costs something: it is called only with checkpointing on.
Return the protocol state this attempt continues from a checkpoint, once, or None.
Mark this episode as a restart: a server it uses cannot capture its part, such as a restart-only agent.
Call it when a seed reply says so (nemo_gym.server_runtime._checkpoint.steps.seed_restarts).
The episode then never holds up a checkpoint and is never exported; after a crash,
the controller starts it over from its input.
Run one protocol step as a wait or replay step (see nemo_gym.server_runtime._checkpoint.steps).
Use replay only for a step that is safe to run again after a crash
and does not change checkpointed state; everything else waits.
The wall-clock bound of this episode: default_episode_timeout_seconds (None is no bound).
A protocol whose tasks carry their own budgets may derive a longer one per episode (the Harbor server does for a multi-step task whose steps add up to more than the default); it must never raise here, as nothing of the episode has started yet.
Adapt runtime event, local policy and evidence to the protocol’s wire failure model.
The failure’s FailureReport goes in the reply’s failure_report, beside the failure.
Run one concrete environment protocol.
Take part in partial-rollout checkpoints when they are enabled.
Every episode is tracked.
A protocol that records boundaries (checkpoint_mode = "exported") continues after a restore;
the episodes of a protocol that records none are restarts,
which never hold up a checkpoint and start over from their input after a crash.
Call this from any setup_webserver that builds its own app.
Bases: BaseRunServerInstanceConfig
Configure protocol-neutral episode limits.
Hold bounded process-local cleanup callbacks for one episode.
Callbacks:
- are process-local Python objects, although they may issue remote close requests;
- run sequentially in LIFO order, each with its own cleanup timeout;
- must be idempotent because a timed-out remote request may have succeeded;
- are lost on process or host failure, so remote owners need expiry or reaping.
Callback failures or timeouts do not prevent later owners from tearing down their resources. Total cleanup is bounded by the number of registered callbacks times the per-callback timeout. Cleanup has no durable retry after this context is discarded.
Close one registered participant at a protocol boundary.
Run this idempotent callback once.
The episode timeout bounds calls made during the protocol. The context’s cleanup timeout bounds callbacks left for final unwinding.
Turn a failed call to another server into the episode’s failure.
failure_reason keeps the dependency’s own explanation from its response body
(see format_exception_with_response_body), prefixed with the error’s class name when name_error_type is set,
and bounded to the failure’s 2000 characters. terminal defaults to
not is_retryable_dependency_error(error); pass it only where the stage decides on its own.
fields retain partial evidence for the response adapter, such as a partial response.
The environment’s handled episode failure: one NgFailureEvent with the local policy kept apart.
terminal and evidence are runtime outcome data for the response adapter, never event fields; the
adapter builds the protocol’s EpisodeFailure after cleanup. failure_kind is the outcome’s kind and
may stay unset; the event then records the registered episode_failed.
Whether error is an environment failure built by handled_episode_failure.
Return whether a failed call to another server is worth another episode attempt.
An HTTP error is classified by its status (see is_retryable_dependency_status). A refused or dropped
connection and a timeout are transient. Any other error, such as a reply that fails validation, would fail the
same way again.
Return whether an HTTP error status from another server is worth another episode attempt.
408, 425, 429 and every 5xx are retried; every other 4xx is terminal.