nemo_gym.harness_capabilities.checker

Deterministic model and semantic checks over native Gym JSON.

Module Contents

Classes

Name	Description
`EvidenceScope`	Applicability of a harness/benchmark pair; absence never implies N/A.
`Finding`	-
`_RecordInspector`	Shared evidence and findings for capability-specific artifact checks.

Functions

Name	Description
`_mapping`	-
`_missing_content`	Reject unresolved media/opaque references; validate inline image data.
`_number`	-
`_objects`	-
`_provider_usage`	-
`_resolve`	-
`_successful`	-
`gate_passes`	All applicable P0 rows and at least one exact call join must pass.
`inspect_record`	Check TE-1–TE-9 on retained JSONL; do not infer missing evidence or applicability.

Data

NAMES

P0

PROFILE

API

class nemo_gym.harness_capabilities.checker.EvidenceScope(
    tools: bool = True,
    verifier: bool = True,
    steps: bool = True
)

Dataclass

Applicability of a harness/benchmark pair; absence never implies N/A.

steps

bool = True

tools

bool = True

verifier

bool = True

class nemo_gym.harness_capabilities.checker.Finding(
    evidence: str,
    assertion: str,
    location: str,
    reason: str
)

Dataclass

assertion

str

evidence

str

location

str

reason

str

class nemo_gym.harness_capabilities.checker._RecordInspector(
    record: dict,
    source: str,
    scope: nemo_gym.harness_capabilities.checker.EvidenceScope
)

Shared evidence and findings for capability-specific artifact checks.

bundle

= _mapping(self.record.get('ng_agent_observations'))

calls

= _objects(self.capture.get('calls'))

capture

= _mapping(self.record.get('ng_model_call_capture'))

findings

list[Finding] = []

invocation_ids

invocation_locations

invocations

not_applicable

set[str] = set()

observations

= _objects(self.bundle.get('records'))

owners

dict[int, list[str]] = {i: [] for i in (range(len(self.calls)))}

tool_locations

tools

trajectory

= _mapping(self.record.get('ng_trajectory'))

turns

= _objects(self.trajectory.get('turns'))

nemo_gym.harness_capabilities.checker._RecordInspector._check_records(
    capability: str,
    records: list[dict],
    location: str
) -> None

nemo_gym.harness_capabilities.checker._RecordInspector._check_token_values(
    values: dict,
    location: str
) -> None

nemo_gym.harness_capabilities.checker._RecordInspector._fail(
    capability: str,
    assertion: str,
    location: str,
    reason: str
) -> None

nemo_gym.harness_capabilities.checker._RecordInspector._require_text(
    capability: str,
    item: dict,
    fields: tuple[str, ...],
    location: str
) -> None

nemo_gym.harness_capabilities.checker._RecordInspector.check_capture_presence() -> None

Missing captures invalidate every capability that depends on calls.

nemo_gym.harness_capabilities.checker._RecordInspector.check_content_references() -> None

TE-4/TE-7: reject unavailable media and opaque payload dependencies.

nemo_gym.harness_capabilities.checker._RecordInspector.check_gaps() -> None

Propagate explicit producer gaps to dependent capabilities.

nemo_gym.harness_capabilities.checker._RecordInspector.check_history() -> None

TE-4: ordered model request and response history.

nemo_gym.harness_capabilities.checker._RecordInspector.check_identity() -> None

Record integrity: readable records, supported schema and consistent identity.

nemo_gym.harness_capabilities.checker._RecordInspector.check_model_calls() -> None

TE-1: faithful terminal outcomes, including failure and cancellation.

nemo_gym.harness_capabilities.checker._RecordInspector.check_ownership() -> None

TE-8: each captured attempt resolves to one explicit invocation owner.

nemo_gym.harness_capabilities.checker._RecordInspector.check_payloads() -> None

TE-7: retained payloads or explicit no-response transport failures.

nemo_gym.harness_capabilities.checker._RecordInspector.check_step_join() -> None

TE-9: exact policy-attempt membership, independent of invocation refs.

nemo_gym.harness_capabilities.checker._RecordInspector.check_token_counts() -> None

TE-2: preserve supplied usage on all attempts; report availability separately.

nemo_gym.harness_capabilities.checker._RecordInspector.check_tools() -> None

TE-5: unique executed tools, request joins and terminal outcomes.

nemo_gym.harness_capabilities.checker._RecordInspector.check_turns() -> None

TE-3 contains step fields; TE-9 independently checks call linkage.

nemo_gym.harness_capabilities.checker._RecordInspector.check_verifier() -> None

TE-6’s shipped Gym surface: rollout verification reward and terminal resolution.

nemo_gym.harness_capabilities.checker._RecordInspector.result() -> dict

Report evidence correctness, metric availability and delivery independently.

nemo_gym.harness_capabilities.checker._mapping(
    value: object
) -> dict

nemo_gym.harness_capabilities.checker._missing_content(
    value: object
) -> bool

Reject unresolved media/opaque references; validate inline image data.

nemo_gym.harness_capabilities.checker._number(
    value: object
) -> typing.TypeGuard[int | float]

nemo_gym.harness_capabilities.checker._objects(
    value: object
) -> list[dict]

nemo_gym.harness_capabilities.checker._provider_usage(
    usage: dict
) -> dict

nemo_gym.harness_capabilities.checker._resolve(
    reference: dict,
    calls: list[dict]
) -> list[int]

nemo_gym.harness_capabilities.checker._successful(
    call: dict
) -> bool

nemo_gym.harness_capabilities.checker.gate_passes(
    evidence: dict
) -> bool

All applicable P0 rows and at least one exact call join must pass.

nemo_gym.harness_capabilities.inspect_record(
    record: dict,
    source: str = 'record',
    scope: nemo_gym.harness_capabilities.checker.EvidenceScope = EvidenceScope()
) -> dict

Check TE-1–TE-9 on retained JSONL; do not infer missing evidence or applicability.

nemo_gym.harness_capabilities.checker.NAMES = {'TE-1': 'model_call_status', 'TE-2': 'token_counts', 'TE-3': 'steps', 'TE-4': '...

nemo_gym.harness_capabilities.checker.P0 = ('TE-1', 'TE-2', 'TE-3', 'TE-4', 'TE-5', 'TE-6', 'TE-7')

nemo_gym.harness_capabilities.checker.PROFILE = 'gym-p0/v1'