nemo_gym.failure_kinds

View as Markdown

Shared names for why a rollout failed.

Gym describes failures with free text and component-local labels, so the same failure arrives at a collector, a log line and a metric under three different names and cannot be grouped. This module holds the one vocabulary those producers share.

A name says what kind of thing went wrong and nothing else. Whether to retry, whether the request may be replayed, whether a completed result should be masked, and what to tell a person are all properties of the occurrence, not of the name — they live on the failure record, on the verify response, and in failure_reason respectively. Keeping them out is deliberate: metadata attached to a name goes stale the moment one caller wants to retry what another caller does not.

Names are low cardinality on purpose. They are safe as a metric label or a span dimension; failure_reason never is, because it carries occurrence detail and can be unbounded.

This module imports only the standard library so that data tooling can read the vocabulary without installing any server’s requirements.

Module Contents

Functions

NameDescription
is_namespacedWhether name is an environment’s own <server>:<kind> extension.
is_registeredWhether name is part of the shared vocabulary.
validate_failure_kindReturn name unchanged, warning once per unregistered value.

Data

AGENT_COMMAND_NOT_ALLOWED

AGENT_REQUEST_FAILED

AGENT_RUN_ERROR

AGENT_TIMEOUT

CANCELLED

COHORT_INCOMPLETE

FAILURE_KINDS

JUDGE_FAILED

JUDGE_UNPARSEABLE

KILL_SHAPED

PERSISTENCE_FAILED

PROVIDER_OOM_KILLED

PROVIDER_QUOTA_EXHAUSTED

PROVIDER_UNAVAILABLE

SESSION_EXPIRED

SESSION_LOST

SESSION_RELEASE_FAILED

SHUTDOWN

TRANSPORT_LOCAL_RESOURCE

TRANSPORT_PEER_DROP

TRANSPORT_TIMEOUT

TRANSPORT_UNREACHABLE

VERIFIER_ERROR

_NAMESPACED_PATTERN

_NAME_PATTERN

_WARNED_UNKNOWN

logger

API

nemo_gym.failure_kinds.is_namespaced(
name: str
) -> bool

Whether name is an environment’s own <server>:<kind> extension.

nemo_gym.failure_kinds.is_registered(
name: str
) -> bool

Whether name is part of the shared vocabulary.

nemo_gym.failure_kinds.validate_failure_kind(
name: str | None
) -> str | None

Return name unchanged, warning once per unregistered value.

Producers call this at their boundary. It warns rather than raises so a component that still emits an old label stays visible during migration — rejecting it outright would replace an observable wrong name with an invisible dropped failure, which is worse than the problem this vocabulary exists to fix.

nemo_gym.failure_kinds.AGENT_COMMAND_NOT_ALLOWED = 'agent_command_not_allowed'
nemo_gym.failure_kinds.AGENT_REQUEST_FAILED = 'agent_request_failed'
nemo_gym.failure_kinds.AGENT_RUN_ERROR = 'agent_run_error'
nemo_gym.failure_kinds.AGENT_TIMEOUT = 'agent_timeout'
nemo_gym.failure_kinds.CANCELLED = 'cancelled'
nemo_gym.failure_kinds.COHORT_INCOMPLETE = 'cohort_incomplete'
nemo_gym.failure_kinds.FAILURE_KINDS: frozenset[str] = frozenset({TRANSPORT_UNREACHABLE, TRANSPORT_PEER_DROP, TRANSPORT_TIMEOUT, TRANSP...
nemo_gym.failure_kinds.JUDGE_FAILED = 'judge_failed'
nemo_gym.failure_kinds.JUDGE_UNPARSEABLE = 'judge_unparseable'
nemo_gym.failure_kinds.KILL_SHAPED = 'kill_shaped'
nemo_gym.failure_kinds.PERSISTENCE_FAILED = 'persistence_failed'
nemo_gym.failure_kinds.PROVIDER_OOM_KILLED = 'provider_oom_killed'
nemo_gym.failure_kinds.PROVIDER_QUOTA_EXHAUSTED = 'provider_quota_exhausted'
nemo_gym.failure_kinds.PROVIDER_UNAVAILABLE = 'provider_unavailable'
nemo_gym.failure_kinds.SESSION_EXPIRED = 'session_expired'
nemo_gym.failure_kinds.SESSION_LOST = 'session_lost'
nemo_gym.failure_kinds.SESSION_RELEASE_FAILED = 'session_release_failed'
nemo_gym.failure_kinds.SHUTDOWN = 'shutdown'
nemo_gym.failure_kinds.TRANSPORT_LOCAL_RESOURCE = 'transport_local_resource'
nemo_gym.failure_kinds.TRANSPORT_PEER_DROP = 'transport_peer_drop'
nemo_gym.failure_kinds.TRANSPORT_TIMEOUT = 'transport_timeout'
nemo_gym.failure_kinds.TRANSPORT_UNREACHABLE = 'transport_unreachable'
nemo_gym.failure_kinds.VERIFIER_ERROR = 'verifier_error'
nemo_gym.failure_kinds._NAMESPACED_PATTERN = re.compile('[a-z][a-z0-9_]*:[a-z][a-z0-9_]*')
nemo_gym.failure_kinds._NAME_PATTERN = re.compile('[a-z][a-z0-9_]*')
nemo_gym.failure_kinds._WARNED_UNKNOWN: set[str] = set()
nemo_gym.failure_kinds.logger = logging.getLogger(__name__)