nemo_gym.server_runtime.verifier_observability

Observe whole verify executions independently of their captured model calls.

Module Contents

Functions

Name	Description
`_finish`	-
`merge_verifier_call_observation`	Link a captured attempt to its caller-declared verifier invocation, when supplied.
`observe_llm_judge_grading`	Execute a reviewed async LLM-judge grading once and record its actual result in the active invocation.
`observe_programmatic_grading`	Execute a reviewed synchronous scorer once and record its actual result in the active invocation.
`observe_verifier`	Return verification evidence with its result, leaving persistence to collection.
`observe_verifier_sandbox`	Bind owner-observed sandbox facts to the active verification.

Data

API

nemo_gym.server_runtime.verifier_observability._finish(
    invocation: nemo_gym.trajectory.trajectory_record.VerifierInvocation,
    result: object,
    verifier_name: str | None,
    graded_in_process: bool
) -> None

nemo_gym.server_runtime.verifier_observability.merge_verifier_call_observation(
    invocations: dict[str, nemo_gym.trajectory.trajectory_record.VerifierInvocation],
    exchange: collections.abc.Mapping[str, object],
    model_call_id: str
) -> None

Link a captured attempt to its caller-declared verifier invocation, when supplied.

nemo_gym.server_runtime.verifier_observability.observe_llm_judge_grading(
    calculate: collections.abc.Callable[[], collections.abc.Awaitable[nemo_gym.server_runtime.verifier_observability.T]]
) -> nemo_gym.server_runtime.verifier_observability.T

async

Execute a reviewed async LLM-judge grading once and record its actual result in the active invocation.

The judge replies parsed by call_judge during calculate become the operation’s dependencies. It completes only when every judge reply was identified, at least one was, the call site marked every reply with record_parsed_judge_verdict and the result binds; otherwise it stays unknown. Errors propagate unchanged.

nemo_gym.server_runtime.verifier_observability.observe_programmatic_grading(
    calculate: collections.abc.Callable[[], nemo_gym.server_runtime.verifier_observability.T]
) -> nemo_gym.server_runtime.verifier_observability.T

Execute a reviewed synchronous scorer once and record its actual result in the active invocation.

This is an explicit call-site contract for pure calculations, not a purity detector for arbitrary callbacks. Judge/SDK/sandbox paths must not use it to label a returned result as programmatic.

nemo_gym.server_runtime.verifier_observability.observe_verifier(
    verify_fn: collections.abc.Callable[nemo_gym.server_runtime.verifier_observability.P, collections.abc.Awaitable[nemo_gym.server_runtime.verifier_observability.T]],
    enabled: bool,
    verifier_name: str | None = None,
    uninstrumented_models: bool = False
) -> collections.abc.Callable[nemo_gym.server_runtime.verifier_observability.P, collections.abc.Awaitable[nemo_gym.server_runtime.verifier_observability.T | starlette.responses.Response]]

Return verification evidence with its result, leaving persistence to collection.

A verify execution named by verifier_name that completes without sending any model request also returns its grading record: the verifier, the score and its verdict, bound to the graded response. A verifier that may ask a model through a client that notes no request (uninstrumented_models) never returns one: missing notes do not show that its grading was model-free.

HTTP failures keep their status and carry evidence in the error body. Cancellation still propagates; a cancelled or lost response cannot deliver in-memory evidence.

nemo_gym.server_runtime.verifier_observability.observe_verifier_sandbox(
    observation: nemo_gym.trajectory.trajectory_record.SandboxObservation | collections.abc.Mapping[str, object]
) -> None

Bind owner-observed sandbox facts to the active verification.

These facts do not certify remote liveness or dependency-free grading.

nemo_gym.server_runtime.verifier_observability.P = ParamSpec('P')

nemo_gym.server_runtime.verifier_observability.T = TypeVar('T')

nemo_gym.server_runtime.verifier_observability._active_verifier: ContextVar[VerifierInvocation | None] = ContextVar('active_verifier_grading', default=None)

nemo_gym.server_runtime.verifier_observability.logger = logging.getLogger(__name__)