nemo_gym.verifier_utils.sandbox_verification

Run a verifier’s grading harness in a fresh, single-use provider sandbox.

A resources server uploads its runner and harness files with a JSON request; inside the sandbox sandbox_runner.py request.json result.json grades the submission and writes a result file, which is downloaded and parsed before the sandbox is stopped. Only the provider-neutral sandbox API is used, so any provider works.

Submission failures (wrong answer, timeout, crash) are the harness’s to report in the result. A failure of the sandbox or the runner raises SandboxVerificationError with a failure_kind so the caller can mask the sample instead of scoring it zero.

Module Contents

Classes

Name	Description
`SandboxVerificationConfig`	Sandbox settings for grading one submission per sandbox.
`SandboxVerificationError`	The sandbox or runner failed, so the sample has no valid reward.

Functions

Name	Description
`_runner_log_tail`	-
`run_verification_sandbox`	Grade one submission in a fresh sandbox and return the parsed result.

Data

API

class nemo_gym.verifier_utils.sandbox_verification.SandboxVerificationConfig()

Bases: BaseModel

Sandbox settings for grading one submission per sandbox.

env

dict[str, str] = Field(default_factory=dict)

image

str | None = None

metadata

dict[str, str] = Field(default_factory=dict)

provider_options

dict[str, Any] = Field(default_factory=dict)

python

str = 'python3'

ready_timeout_s

float = 600

resources

SandboxResources = Field(default_factory=SandboxResources)

nemo_gym.verifier_utils.sandbox_verification.SandboxVerificationConfig._parse_resources(
    value: object
) -> object

classmethod

class nemo_gym.verifier_utils.sandbox_verification.SandboxVerificationError(
    failure_kind: str,
    message: str
)

Exception

Bases: Exception

The sandbox or runner failed, so the sample has no valid reward.

nemo_gym.verifier_utils.sandbox_verification._runner_log_tail(
    sandbox: nemo_gym.sandbox.api.AsyncSandbox,
    work_dir: str
) -> str

async

nemo_gym.verifier_utils.sandbox_verification.run_verification_sandbox(
    provider: collections.abc.Mapping[str, typing.Any] | nemo_gym.sandbox.providers.SandboxProvider,
    config: nemo_gym.verifier_utils.sandbox_verification.SandboxVerificationConfig,
    files: collections.abc.Mapping[str, str],
    request: collections.abc.Mapping[str, typing.Any],
    exec_timeout_s: float,
    parse_result: collections.abc.Callable[[str], nemo_gym.verifier_utils.sandbox_verification.T],
    metadata: collections.abc.Mapping[str, str],
    work_dir: str
) -> nemo_gym.verifier_utils.sandbox_verification.T

async

Grade one submission in a fresh sandbox and return the parsed result.

Parameters:

provider

Mapping[str, Any] | SandboxProvider

Mapping[str, Any] | SandboxProvider — Provider instance or provider config mapping.

config

SandboxVerificationConfig

SandboxVerificationConfig — Image, resources and provider settings.

files

Mapping[str, str]

Text files to upload, keyed by path relative to work_dir. Must include sandbox_runner.py, which is run as sandbox_runner.py request.json result.json.

request

Mapping[str, Any]

JSON-serializable request written to request.json. It stays in the runner’s working directory, so anything the runner launches can read it: keep expected outputs out of it or delete it before running the submission.

exec_timeout_s

float

Deadline for the runner.

parse_result

Callable[[str], T]

Callable[[str], T ] — Parses and validates the text of result.json. Any exception it raises is reported as a verifier_error.

metadata

Mapping[str, str]

Sandbox metadata; config.metadata entries win on conflicts.

work_dir

str

Absolute directory in the sandbox; each call uses its own subdirectory of it for the uploaded files and the result.

Raises:

SandboxVerificationError: provider_unavailable if the sandbox could not start, verifier_error if the runner failed or its result is invalid.

nemo_gym.verifier_utils.sandbox_verification.LOG = logging.getLogger(__name__)

nemo_gym.verifier_utils.sandbox_verification.RUNNER_FILE = 'sandbox_runner.py'

nemo_gym.verifier_utils.sandbox_verification.T = TypeVar('T')

nemo_gym.verifier_utils.sandbox_verification._LOG_TAIL_BYTES = 2000

nemo_gym.verifier_utils.sandbox_verification._TTL_MARGIN_S = 300