API ReferenceFull Library ReferenceNemo GymNemo Gym

nemo_gym.config_types

View as Markdown

Module Contents

Classes

NameDescription
AgentServerRef-
AggregateMetricsResponse from /aggregate_metrics.
AggregateMetricsRequestPOST body for /aggregate_metrics.
AlmostServerErrorOne or more server blocks are almost-servers (right shape, failed validation) and
BaseNeMoGymCLIConfig-
BaseRunServerConfig-
BaseRunServerInstanceConfig-
BaseRunServerTypeConfig-
BaseServerConfig-
BaseServerInstanceConfig-
BaseServerTypeConfig-
BaseUploadJsonlDatasetHuggingFaceConfigUpload a JSONL dataset to HuggingFace Hub with automatic naming based on domain and resources server.
BenchmarkDatasetConfig-
ConfigErrorBase for user-facing configuration errors.
ConfigMissingValuesErrorOne or more required config values are still unset (OmegaConf ’???’) after merging.
ConfigPathNotFoundErrorA config_paths entry could not be found in the cwd or the Gym install location.
DatasetConfig-
DeleteJsonlDatasetGitlabConfigDelete a dataset from GitLab Model Registry (prompts for confirmation).
DomainThe capability a resources server primarily evaluates or trains.
DownloadJsonlDatasetGitlabConfigDownload a JSONL dataset from GitLab Model Registry.
DownloadJsonlDatasetHuggingFaceConfigDownload a JSONL dataset from HuggingFace Hub to local filesystem.
GitlabDatasetSourceUnified source: for a dataset fetched from the GitLab model registry.
HuggingFaceDatasetSourceUnified source: for a dataset fetched from the HuggingFace Hub.
InheritPathNotFoundErrorAn _inherit_from / swap / copy directive references a config path that does not exist.
JsonlDatasetGitlabIdentifer-
JsonlDatasetHuggingFaceIdentifer-
MalformedConfigPathsErrorconfig_paths was not a list of paths (e.g. a scalar string was passed).
ModelServerRef-
NoServerInstancesErrorA run was requested but the merged config defines no server instances to start.
ResourcesServerInstanceConfig-
ResourcesServerRef-
ResourcesServerTypeConfig-
ResponsesAPIAgentServerInstanceConfig-
ResponsesAPIAgentServerTypeConfig-
ResponsesAPIModelServerInstanceConfig-
ResponsesAPIModelServerTypeConfig-
ServerRefNotFoundErrorA server cross-reference points to an instance that is not defined in the merged config.
UploadJsonlDatasetGitlabConfigUpload a local jsonl dataset artifact to Gitlab.
UploadJsonlDatasetHuggingFaceConfigUpload a JSONL dataset to HuggingFace Hub and automatically delete from GitLab after successful upload.
UploadJsonlDatasetHuggingFaceMaybeDeleteConfigUpload a JSONL dataset to HuggingFace Hub with optional GitLab deletion after successful upload.
WANDBConfig-

Functions

NameDescription
is_almost_serverDetects if a config looks like a server but might fail validation.
is_server_ref-
maybe_get_server_instance_configReturns ServerInstanceConfig if a valid server, otherwise None with error details

Data

AGENT_REF_KEY

DatasetSource

DatasetType

ServerInstanceConfig

ServerInstanceConfigTypeAdapter

ServerRef

ServerRefTypeAdapter

ServerTypeConfig

API

class nemo_gym.config_types.AgentServerRef()

Bases: BaseModel

name
str
type
Literal['responses_api_agents']
class nemo_gym.config_types.AggregateMetrics()

Bases: BaseModel

Response from /aggregate_metrics.

Flat string keys for direct logging to W&B/MLflow.

agent_metrics
Dict[str, Any]
group_level_metrics
List[Dict[str, Any]]
key_metrics
Dict[str, Any]
class nemo_gym.config_types.AggregateMetricsRequest()

Bases: BaseModel

POST body for /aggregate_metrics.

Each item is a stripped verify response dict containing at minimum:

  • TASK_INDEX_KEY_NAME: int
  • “reward”: float
verify_responses
List[Dict[str, Any]]
class nemo_gym.config_types.AlmostServerError()

Bases: ConfigError, ValueError

One or more server blocks are almost-servers (right shape, failed validation) and error_on_almost_servers is set, so the run is aborted.

class nemo_gym.config_types.BaseNeMoGymCLIConfig()

Bases: BaseModel

nemo_gym.config_types.BaseNeMoGymCLIConfig.pre_process(
    data
)
classmethod
class nemo_gym.config_types.BaseRunServerConfig()

Bases: BaseServerConfig

domain
Optional[Domain] = None
entrypoint
str
class nemo_gym.config_types.BaseRunServerInstanceConfig()

Bases: BaseRunServerConfig

name
str
class nemo_gym.config_types.BaseRunServerTypeConfig()

Bases: BaseRunServerConfig

datasets
Optional[List[Union[DatasetConfig, BenchmarkDatasetConfig]]] = None
host
Optional[str] = None
model_config
= ConfigDict(extra='allow')
port
Optional[int] = None
class nemo_gym.config_types.BaseServerConfig()

Bases: BaseModel

host
str
num_workers
Optional[int] = None
port
int
class nemo_gym.config_types.BaseServerInstanceConfig()

Bases: BaseServerTypeConfig

datasets
Optional[List[Union[DatasetConfig, BenchmarkDatasetConfig]]]
model_config
= ConfigDict(arbitrary_types_allowed=True)
name
str
server_type_config_dict
DictConfig = Field(exclude=True)
nemo_gym.config_types.BaseServerInstanceConfig.get_inner_run_server_config() -> nemo_gym.config_types.BaseRunServerTypeConfig
nemo_gym.config_types.BaseServerInstanceConfig.get_inner_run_server_config_dict() -> omegaconf.DictConfig
nemo_gym.config_types.BaseServerInstanceConfig.get_server_ref() -> nemo_gym.config_types.ServerRef
class nemo_gym.config_types.BaseServerTypeConfig()

Bases: BaseModel

SERVER_TYPE
Union[Literal['responses_api_models'], Literal['resources_servers'], Literal['responses_api_agents']]
class nemo_gym.config_types.BaseUploadJsonlDatasetHuggingFaceConfig()

Bases: BaseNeMoGymCLIConfig

Upload a JSONL dataset to HuggingFace Hub with automatic naming based on domain and resources server.

Examples:

resource_config_path="resources_servers/example_multi_step/configs/example_multi_step.yaml"
gym dataset upload         +dataset_name=my_dataset         +input_jsonl_fpath=data/train.jsonl         +resource_config_path=${resource_config_path}
commit_description
Optional[str]
commit_message
Optional[str]
create_pr
bool
dataset_name
Optional[str]
hf_collection_name
str
hf_collection_slug
str
hf_dataset_prefix
str
hf_organization
str
hf_token
str
input_jsonl_fpath
str
resource_config_path
str
revision
Optional[str]
split
Literal['train', 'validation', 'test']
class nemo_gym.config_types.BenchmarkDatasetConfig()

Bases: BaseModel

jsonl_fpath
Path
name
str
num_repeats
int = Field(default=1, ge=1)
prepare_script
Path
prompt_config
Optional[Path]
type
Literal['benchmark']
class nemo_gym.config_types.ConfigError()
Exception

Bases: Exception

Base for user-facing configuration errors.

These represent actionable user mistakes (typos, missing files, malformed input) rather than internal bugs. The CLI catches ConfigError and prints just the message — no traceback — while still leaving them as ordinary exceptions so callers like validate can catch and format them.

class nemo_gym.config_types.ConfigMissingValuesError()

Bases: ConfigError, ValueError

One or more required config values are still unset (OmegaConf ’???’) after merging.

class nemo_gym.config_types.ConfigPathNotFoundError()

Bases: ConfigError, FileNotFoundError

A config_paths entry could not be found in the cwd or the Gym install location.

class nemo_gym.config_types.DatasetConfig()

Bases: BaseModel

gitlab_identifier
Optional[JsonlDatasetGitlabIdentifer] = None
huggingface_identifier
Optional[JsonlDatasetHuggingFaceIdentifer] = None
jsonl_fpath
str
license
Optional[Union[Literal['Apache 2.0'], Literal['MIT'], Literal['Creative Commons Attribution 4.0 International'], Literal['Creative Commons Attribution-ShareAlike 4.0 International'], Literal['NVIDIA Internal Use Only, Do Not Distribute'], Literal['NVIDIA Evaluation Dataset License Agreement'], Literal['TBD'], Literal['GNU General Public License v3.0']]] = None
name
str
num_repeats
int = Field(default=1, ge=1)
source
Optional[DatasetSource] = None
type
DatasetType

Reconcile the unified source: with the legacy *_identifier fields.

The unified source: block is mutually exclusive with the legacy identifiers. The two legacy identifiers may still be set together (a gitlab-primary / huggingface-fallback pair selected at download time by config.data_source) for backward compatibility. A legacy identifier emits a deprecation warning and, when a single backend is given, is mirrored into source; conversely a source: is mirrored back into the matching legacy field so existing consumers that read gitlab_identifier/huggingface_identifier keep working.

class nemo_gym.config_types.DeleteJsonlDatasetGitlabConfig()

Bases: BaseNeMoGymCLIConfig

Delete a dataset from GitLab Model Registry (prompts for confirmation).

Examples:

gym dataset rm +dataset_name=old_dataset
dataset_name
str
class nemo_gym.config_types.Domain

Bases: enum.Enum

The capability a resources server primarily evaluates or trains.

Pick the single domain that best fits the task. If several seem to apply, choose the most specific one (e.g. prefer math or coding over agent); use other only when none of the specific values fit. The values:

  • math — mathematical problem solving (e.g. AIME, MATH, GSM8K).
  • coding — code generation, repair, or execution (e.g. SWE-bench, LiveCodeBench).
  • agent — multi-step, tool-using / environment-interacting tasks (e.g. tau2, workplace_assistant). Prefer a more specific value when the task is really math/coding/etc.
  • knowledge — factual or domain-knowledge question answering (e.g. GPQA, MMLU).
  • instruction_following — adherence to explicit formatting/constraints (e.g. IFEval).
  • long_context — reasoning over long inputs (e.g. RULER, long-document QA).
  • safety — refusing harmful content / resisting jailbreaks & prompt injection.
  • games — interactive game environments (e.g. blackjack, tetris).
  • translation — machine translation quality (e.g. WMT).
  • e2e — end-to-end pipelines spanning multiple capabilities at once.
  • rlhf — preference / reward-model / LLM-as-judge evaluations.
  • other — catch-all when no specific domain above applies.
AGENT
= 'agent'
CODING
= 'coding'
E2E
= 'e2e'
GAMES
= 'games'
INSTRUCTION_FOLLOWING
= 'instruction_following'
KNOWLEDGE
= 'knowledge'
LONG_CONTEXT
= 'long_context'
MATH
= 'math'
OTHER
= 'other'
RLHF
= 'rlhf'
SAFETY
= 'safety'
TRANSLATION
= 'translation'
class nemo_gym.config_types.DownloadJsonlDatasetGitlabConfig()

Bases: JsonlDatasetGitlabIdentifer, BaseNeMoGymCLIConfig

Download a JSONL dataset from GitLab Model Registry.

Examples:

gym dataset download --storage gitlab         +dataset_name=example_multi_step         +version=0.0.1         +artifact_fpath=train.jsonl         +output_fpath=data/train.jsonl
artifact_fpath
str
dataset_name
str = Field(description='The dataset name.')
output_fpath
str
version
str
class nemo_gym.config_types.DownloadJsonlDatasetHuggingFaceConfig()

Bases: JsonlDatasetHuggingFaceIdentifer, BaseNeMoGymCLIConfig

Download a JSONL dataset from HuggingFace Hub to local filesystem.

Examples:

gym dataset download         +repo_id=NVIDIA/NeMo-Gym-Math-example_multi_step-v1         +artifact_fpath=train.jsonl         +output_fpath=data/train.jsonl
hf_token
Optional[str]
output_dirpath
Optional[str]
output_fpath
Optional[str]
split
Optional[Literal['train', 'validation', 'test']]
class nemo_gym.config_types.GitlabDatasetSource()

Bases: BaseModel

Unified source: for a dataset fetched from the GitLab model registry.

artifact_fpath
str
dataset_name
str
type
Literal['gitlab']
version
str
class nemo_gym.config_types.HuggingFaceDatasetSource()

Bases: BaseModel

Unified source: for a dataset fetched from the HuggingFace Hub.

artifact_fpath
Optional[str] = None
repo_id
str
type
Literal['huggingface']
class nemo_gym.config_types.InheritPathNotFoundError()

Bases: ConfigError, ValueError

An _inherit_from / swap / copy directive references a config path that does not exist.

class nemo_gym.config_types.JsonlDatasetGitlabIdentifer()

Bases: BaseModel

artifact_fpath
str
dataset_name
str
version
str
class nemo_gym.config_types.JsonlDatasetHuggingFaceIdentifer()

Bases: BaseModel

artifact_fpath
Optional[str]
repo_id
str = Field(description='The repo id.')
class nemo_gym.config_types.MalformedConfigPathsError()

Bases: ConfigError, ValueError

config_paths was not a list of paths (e.g. a scalar string was passed).

class nemo_gym.config_types.ModelServerRef()

Bases: BaseModel

name
str
type
Literal['responses_api_models']
class nemo_gym.config_types.NoServerInstancesError()

Bases: ConfigError, ValueError

A run was requested but the merged config defines no server instances to start.

class nemo_gym.config_types.ResourcesServerInstanceConfig()

Bases: ResourcesServerTypeConfig, BaseServerInstanceConfig

class nemo_gym.config_types.ResourcesServerRef()

Bases: BaseModel

name
str
type
Literal['resources_servers']
class nemo_gym.config_types.ResourcesServerTypeConfig()

Bases: BaseServerTypeConfig

SERVER_TYPE
Literal['resources_servers'] = 'resources_servers'
model_config
= ConfigDict(extra='allow')
resources_servers
Dict[str, BaseRunServerTypeConfig] = Field(min_length=1, max_length=1)
class nemo_gym.config_types.ResponsesAPIAgentServerInstanceConfig()

Bases: ResponsesAPIAgentServerTypeConfig, BaseServerInstanceConfig

class nemo_gym.config_types.ResponsesAPIAgentServerTypeConfig()

Bases: BaseServerTypeConfig

SERVER_TYPE
Literal['responses_api_agents'] = 'responses_api_agents'
model_config
= ConfigDict(extra='allow')
responses_api_agents
Dict[str, BaseRunServerTypeConfig] = Field(min_length=1, max_length=1)
class nemo_gym.config_types.ResponsesAPIModelServerInstanceConfig()

Bases: ResponsesAPIModelServerTypeConfig, BaseServerInstanceConfig

class nemo_gym.config_types.ResponsesAPIModelServerTypeConfig()

Bases: BaseServerTypeConfig

SERVER_TYPE
Literal['responses_api_models'] = 'responses_api_models'
model_config
= ConfigDict(extra='allow')
responses_api_models
Dict[str, BaseRunServerTypeConfig] = Field(min_length=1, max_length=1)
class nemo_gym.config_types.ServerRefNotFoundError()

Bases: ConfigError, ValueError

A server cross-reference points to an instance that is not defined in the merged config.

class nemo_gym.config_types.UploadJsonlDatasetGitlabConfig()

Bases: BaseNeMoGymCLIConfig

Upload a local jsonl dataset artifact to Gitlab.

Examples:

gym dataset upload --storage gitlab         +dataset_name=example_multi_step         +version=0.0.1         +input_jsonl_fpath=data/train.jsonl
dataset_name
str = Field(description='The dataset name.')
input_jsonl_fpath
str
version
str
class nemo_gym.config_types.UploadJsonlDatasetHuggingFaceConfig()

Bases: BaseUploadJsonlDatasetHuggingFaceConfig

Upload a JSONL dataset to HuggingFace Hub and automatically delete from GitLab after successful upload.

This command always deletes the dataset from GitLab after uploading to HuggingFace. Use gym dataset upload if you want optional deletion control.

Examples:

resource_config_path="resources_servers/example_multi_step/configs/example_multi_step.yaml"
gym dataset migrate         +dataset_name=my_dataset         +input_jsonl_fpath=data/train.jsonl         +resource_config_path=${resource_config_path}
forbidden_fields
Set[str] = {'delete_from_gitlab'}
nemo_gym.config_types.UploadJsonlDatasetHuggingFaceConfig.check_forbidden_fields(
    data
)
class nemo_gym.config_types.UploadJsonlDatasetHuggingFaceMaybeDeleteConfig()

Bases: BaseUploadJsonlDatasetHuggingFaceConfig

Upload a JSONL dataset to HuggingFace Hub with optional GitLab deletion after successful upload.

Examples:

resource_config_path="resources_servers/example_multi_step/configs/example_multi_step.yaml"
gym dataset upload         +dataset_name=my_dataset         +input_jsonl_fpath=data/train.jsonl         +resource_config_path=${resource_config_path}         +delete_from_gitlab=true
delete_from_gitlab
Optional[bool]
class nemo_gym.config_types.WANDBConfig()

Bases: BaseModel

is_available
bool
wandb_api_key
Optional[str] = None
wandb_name
Optional[str] = None
wandb_project
Optional[str] = None
nemo_gym.config_types.is_almost_server(
    server_type_config_dict: typing.Any
) -> bool

Detects if a config looks like a server but might fail validation.

nemo_gym.config_types.is_server_ref(
    config_dict: omegaconf.DictConfig
) -> typing.Optional[nemo_gym.config_types.ServerRef]
 
nemo_gym.config_types.maybe_get_server_instance_config(
    name: str,
    server_type_config_dict: typing.Any
) -> typing.Tuple[typing.Optional[nemo_gym.config_types.ServerInstanceConfig], typing.Optional[pydantic.ValidationError]]

Returns ServerInstanceConfig if a valid server, otherwise None with error details

nemo_gym.config_types.AGENT_REF_KEY = 'agent_ref'
 
nemo_gym.config_types.DatasetSource = Annotated[Union[GitlabDatasetSource, HuggingFaceDatasetSource], Field(discrimina...
 
nemo_gym.config_types.DatasetType = Union[Literal['train'], Literal['validation'], Literal['example']]
 
nemo_gym.config_types.ServerInstanceConfig = Union[ResponsesAPIModelServerInstanceConfig, ResourcesServerInstanceConfig, Resp...
 
nemo_gym.config_types.ServerInstanceConfigTypeAdapter = TypeAdapter(ServerInstanceConfig)
 
nemo_gym.config_types.ServerRef = Union[ModelServerRef, ResourcesServerRef, AgentServerRef]
 
nemo_gym.config_types.ServerRefTypeAdapter = TypeAdapter(ServerRef)
 
nemo_gym.config_types.ServerTypeConfig = Union[ResponsesAPIModelServerTypeConfig, ResourcesServerTypeConfig, ResponsesAPI...