`nemo_gym.config_types`#

Module Contents#

Classes#

`BaseNeMoGymCLIConfig`
`ModelServerRef`
`ResourcesServerRef`
`AgentServerRef`
`UploadJsonlDatasetGitlabConfig`	Upload a local jsonl dataset artifact to Gitlab.
`JsonlDatasetGitlabIdentifer`
`DownloadJsonlDatasetGitlabConfig`	Download a JSONL dataset from GitLab Model Registry.
`DeleteJsonlDatasetGitlabConfig`	Delete a dataset from GitLab Model Registry (prompts for confirmation).
`JsonlDatasetHuggingFaceIdentifer`
`BaseUploadJsonlDatasetHuggingFaceConfig`	Upload a JSONL dataset to HuggingFace Hub with automatic naming based on domain and resource server.
`UploadJsonlDatasetHuggingFaceConfig`	Upload a JSONL dataset to HuggingFace Hub and automatically delete from GitLab after successful upload.
`UploadJsonlDatasetHuggingFaceMaybeDeleteConfig`	Upload a JSONL dataset to HuggingFace Hub with optional GitLab deletion after successful upload.
`DownloadJsonlDatasetHuggingFaceConfig`	Download a JSONL dataset from HuggingFace Hub to local filesystem.
`DatasetConfig`
`Domain`
`BaseServerConfig`
`BaseRunServerConfig`
`BaseRunServerInstanceConfig`
`BaseRunServerTypeConfig`
`BaseServerTypeConfig`
`ResponsesAPIModelServerTypeConfig`
`ResourcesServerTypeConfig`
`ResponsesAPIAgentServerTypeConfig`
`BaseServerInstanceConfig`
`ResponsesAPIModelServerInstanceConfig`
`ResourcesServerInstanceConfig`
`ResponsesAPIAgentServerInstanceConfig`

Functions#

`is_server_ref`
`maybe_get_server_instance_config`	Returns ServerInstanceConfig if a valid server, otherwise None with error details
`is_almost_server`	Detects if a config looks like a server but might fail validation.

Data#

`ServerRef`
`ServerRefTypeAdapter`
`DatasetType`
`ServerTypeConfig`
`ServerInstanceConfig`
`ServerInstanceConfigTypeAdapter`
`AGENT_REF_KEY`

API#

class nemo_gym.config_types.BaseNeMoGymCLIConfig(/, **data: typing.Any)[source]#

Bases: pydantic.BaseModel

classmethod pre_process(data)[source]#

class nemo_gym.config_types.ModelServerRef(/, **data: typing.Any)[source]#

Bases: pydantic.BaseModel

type: Literal[responses_api_models]#: None

name: str#: None

class nemo_gym.config_types.ResourcesServerRef(/, **data: typing.Any)[source]#

Bases: pydantic.BaseModel

type: Literal[resources_servers]#: None

name: str#: None

class nemo_gym.config_types.AgentServerRef(/, **data: typing.Any)[source]#

Bases: pydantic.BaseModel

type: Literal[responses_api_agents]#: None

name: str#: None

nemo_gym.config_types.ServerRef#: None

nemo_gym.config_types.ServerRefTypeAdapter#: ‘TypeAdapter(…)’

nemo_gym.config_types.is_server_ref( config_dict: omegaconf.DictConfig, ) → Optional[nemo_gym.config_types.ServerRef][source]#

class nemo_gym.config_types.UploadJsonlDatasetGitlabConfig(/, **data: typing.Any)[source]#

Bases: nemo_gym.config_types.BaseNeMoGymCLIConfig

Upload a local jsonl dataset artifact to Gitlab.

Examples:

ng_upload_dataset_to_gitlab         +dataset_name=example_multi_step         +version=0.0.1         +input_jsonl_fpath=data/train.jsonl

Initialization

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

dataset_name: str#: ‘Field(…)’

version: str#: ‘Field(…)’

input_jsonl_fpath: str#: ‘Field(…)’

class nemo_gym.config_types.JsonlDatasetGitlabIdentifer(/, **data: typing.Any)[source]#

Bases: pydantic.BaseModel

dataset_name: str#: None

version: str#: None

artifact_fpath: str#: None

class nemo_gym.config_types.DownloadJsonlDatasetGitlabConfig(/, **data: typing.Any)[source]#

Bases: nemo_gym.config_types.JsonlDatasetGitlabIdentifer, nemo_gym.config_types.BaseNeMoGymCLIConfig

Download a JSONL dataset from GitLab Model Registry.

Examples:

ng_download_dataset_from_gitlab         +dataset_name=example_multi_step         +version=0.0.1         +artifact_fpath=train.jsonl         +output_fpath=data/train.jsonl

Initialization

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

dataset_name: str#: ‘Field(…)’

version: str#: ‘Field(…)’

artifact_fpath: str#: ‘Field(…)’

output_fpath: str#: ‘Field(…)’

class nemo_gym.config_types.DeleteJsonlDatasetGitlabConfig(/, **data: typing.Any)[source]#

Bases: nemo_gym.config_types.BaseNeMoGymCLIConfig

Delete a dataset from GitLab Model Registry (prompts for confirmation).

Examples:

ng_delete_dataset_from_gitlab +dataset_name=old_dataset

Initialization

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

dataset_name: str#: ‘Field(…)’

class nemo_gym.config_types.JsonlDatasetHuggingFaceIdentifer(/, **data: typing.Any)[source]#

Bases: pydantic.BaseModel

repo_id: str#: ‘Field(…)’

artifact_fpath: Optional[str]#: ‘Field(…)’

class nemo_gym.config_types.BaseUploadJsonlDatasetHuggingFaceConfig(/, **data: typing.Any)[source]#

Bases: nemo_gym.config_types.BaseNeMoGymCLIConfig

Upload a JSONL dataset to HuggingFace Hub with automatic naming based on domain and resource server.

Examples:

resource_config_path="resources_servers/example_multi_step/configs/example_multi_step.yaml"
ng_upload_dataset_to_hf         +dataset_name=my_dataset         +input_jsonl_fpath=data/train.jsonl         +resource_config_path=${resource_config_path}

Initialization

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

hf_token: str#: ‘Field(…)’

hf_organization: str#: ‘Field(…)’

hf_collection_name: str#: ‘Field(…)’

hf_collection_slug: str#: ‘Field(…)’

dataset_name: Optional[str]#: ‘Field(…)’

input_jsonl_fpath: str#: ‘Field(…)’

resource_config_path: str#: ‘Field(…)’

hf_dataset_prefix: str#: ‘Field(…)’

split: Literal[train, validation, test]#: ‘Field(…)’

create_pr: bool#: ‘Field(…)’

revision: Optional[str]#: ‘Field(…)’

commit_message: Optional[str]#: ‘Field(…)’

commit_description: Optional[str]#: ‘Field(…)’

class nemo_gym.config_types.UploadJsonlDatasetHuggingFaceConfig(/, **data: typing.Any)[source]#

Bases: nemo_gym.config_types.BaseUploadJsonlDatasetHuggingFaceConfig

Upload a JSONL dataset to HuggingFace Hub and automatically delete from GitLab after successful upload.

This command always deletes the dataset from GitLab after uploading to HuggingFace. Use ng_upload_dataset_to_hf if you want optional deletion control.

Examples:

resource_config_path="resources_servers/example_multi_step/configs/example_multi_step.yaml"
ng_gitlab_to_hf_dataset         +dataset_name=my_dataset         +input_jsonl_fpath=data/train.jsonl         +resource_config_path=${resource_config_path}

Initialization

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

forbidden_fields: ClassVar[Set[str]]#: None

check_forbidden_fields(data)[source]#

class nemo_gym.config_types.UploadJsonlDatasetHuggingFaceMaybeDeleteConfig(/, **data: typing.Any)[source]#

Bases: nemo_gym.config_types.BaseUploadJsonlDatasetHuggingFaceConfig

Upload a JSONL dataset to HuggingFace Hub with optional GitLab deletion after successful upload.

Examples:

resource_config_path="resources_servers/example_multi_step/configs/example_multi_step.yaml"
ng_upload_dataset_to_hf         +dataset_name=my_dataset         +input_jsonl_fpath=data/train.jsonl         +resource_config_path=${resource_config_path}         +delete_from_gitlab=true

Initialization

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

delete_from_gitlab: Optional[bool]#: ‘Field(…)’

class nemo_gym.config_types.DownloadJsonlDatasetHuggingFaceConfig(/, **data: typing.Any)[source]#

Bases: nemo_gym.config_types.JsonlDatasetHuggingFaceIdentifer, nemo_gym.config_types.BaseNeMoGymCLIConfig

Download a JSONL dataset from HuggingFace Hub to local filesystem.

Examples:

ng_download_dataset_from_hf         +repo_id=NVIDIA/NeMo-Gym-Math-example_multi_step-v1         +artifact_fpath=train.jsonl         +output_fpath=data/train.jsonl

Initialization

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

output_dirpath: Optional[str]#: ‘Field(…)’

output_fpath: Optional[str]#: ‘Field(…)’

hf_token: Optional[str]#: ‘Field(…)’

split: Optional[Literal[train, validation, test]]#: ‘Field(…)’

check_output_path() → nemo_gym.config_types.DownloadJsonlDatasetHuggingFaceConfig[source]#

nemo_gym.config_types.DatasetType#: None

class nemo_gym.config_types.DatasetConfig(/, **data: typing.Any)[source]#

Bases: pydantic.BaseModel

name: str#: None

type: nemo_gym.config_types.DatasetType#: None

jsonl_fpath: str#: None

num_repeats: int#: ‘Field(…)’

gitlab_identifier: Optional[nemo_gym.config_types.JsonlDatasetGitlabIdentifer]#: None

huggingface_identifier: Optional[nemo_gym.config_types.JsonlDatasetHuggingFaceIdentifer]#: None

license: Optional[typing.Union[typing.Literal[Apache 2.0], typing.Literal[MIT], typing.Literal[Creative Commons Attribution 4.0 International], typing.Literal[Creative Commons Attribution-ShareAlike 4.0 International], typing.Literal[NVIDIA Internal Use Only, Do Not Distribute], typing.Literal[TBD]]]#: None

check_train_validation_sets() → nemo_gym.config_types.DatasetConfig[source]#

class nemo_gym.config_types.Domain[source]#

Bases: str, enum.Enum

MATH#: ‘math’

CODING#: ‘coding’

AGENT#: ‘agent’

KNOWLEDGE#: ‘knowledge’

INSTRUCTION_FOLLOWING#: ‘instruction_following’

LONG_CONTEXT#: ‘long_context’

SAFETY#: ‘safety’

GAMES#: ‘games’

TRANSLATION#: ‘translation’

E2E#: ‘e2e’

OTHER#: ‘other’

class nemo_gym.config_types.BaseServerConfig(/, **data: typing.Any)[source]#

Bases: pydantic.BaseModel

host: str#: None

port: int#: None

class nemo_gym.config_types.BaseRunServerConfig(/, **data: typing.Any)[source]#

Bases: nemo_gym.config_types.BaseServerConfig

entrypoint: str#: None

domain: Optional[nemo_gym.config_types.Domain]#: None

class nemo_gym.config_types.BaseRunServerInstanceConfig(/, **data: typing.Any)[source]#

Bases: nemo_gym.config_types.BaseRunServerConfig

name: str#: None

class nemo_gym.config_types.BaseRunServerTypeConfig(/, **data: typing.Any)[source]#

Bases: nemo_gym.config_types.BaseRunServerConfig

model_config#: ‘ConfigDict(…)’

host: Optional[str]#: None

port: Optional[int]#: None

datasets: Optional[List[nemo_gym.config_types.DatasetConfig]]#: None

class nemo_gym.config_types.BaseServerTypeConfig(/, **data: typing.Any)[source]#

Bases: pydantic.BaseModel

SERVER_TYPE: ClassVar[Union[Literal[responses_api_models], Literal[resources_servers], Literal[responses_api_agents]]]#: None

class nemo_gym.config_types.ResponsesAPIModelServerTypeConfig(/, **data: typing.Any)[source]#

Bases: nemo_gym.config_types.BaseServerTypeConfig

SERVER_TYPE: ClassVar[Literal[nemo_gym.config_types.ResponsesAPIModelServerTypeConfig.responses_api_models]]#: ‘responses_api_models’

model_config#: ‘ConfigDict(…)’

responses_api_models: Dict[str, nemo_gym.config_types.BaseRunServerTypeConfig]#: ‘Field(…)’

class nemo_gym.config_types.ResourcesServerTypeConfig(/, **data: typing.Any)[source]#

Bases: nemo_gym.config_types.BaseServerTypeConfig

SERVER_TYPE: ClassVar[Literal[nemo_gym.config_types.ResourcesServerTypeConfig.resources_servers]]#: ‘resources_servers’

model_config#: ‘ConfigDict(…)’

resources_servers: Dict[str, nemo_gym.config_types.BaseRunServerTypeConfig]#: ‘Field(…)’

class nemo_gym.config_types.ResponsesAPIAgentServerTypeConfig(/, **data: typing.Any)[source]#

Bases: nemo_gym.config_types.BaseServerTypeConfig

SERVER_TYPE: ClassVar[Literal[nemo_gym.config_types.ResponsesAPIAgentServerTypeConfig.responses_api_agents]]#: ‘responses_api_agents’

model_config#: ‘ConfigDict(…)’

responses_api_agents: Dict[str, nemo_gym.config_types.BaseRunServerTypeConfig]#: ‘Field(…)’

nemo_gym.config_types.ServerTypeConfig#: None

class nemo_gym.config_types.BaseServerInstanceConfig(/, **data: typing.Any)[source]#

Bases: nemo_gym.config_types.BaseServerTypeConfig

model_config#: ‘ConfigDict(…)’

name: str#: None

server_type_config_dict: omegaconf.DictConfig#: ‘Field(…)’

validate_domain_for_resource_server() → nemo_gym.config_types.BaseServerInstanceConfig[source]#

get_server_ref() → nemo_gym.config_types.ServerRef[source]#

get_inner_run_server_config_dict() → omegaconf.DictConfig[source]#

get_inner_run_server_config() → nemo_gym.config_types.BaseRunServerTypeConfig[source]#

property datasets: Optional[List[nemo_gym.config_types.DatasetConfig]]#

class nemo_gym.config_types.ResponsesAPIModelServerInstanceConfig(/, **data: typing.Any)[source]#: Bases: nemo_gym.config_types.ResponsesAPIModelServerTypeConfig, nemo_gym.config_types.BaseServerInstanceConfig

class nemo_gym.config_types.ResourcesServerInstanceConfig(/, **data: typing.Any)[source]#: Bases: nemo_gym.config_types.ResourcesServerTypeConfig, nemo_gym.config_types.BaseServerInstanceConfig

class nemo_gym.config_types.ResponsesAPIAgentServerInstanceConfig(/, **data: typing.Any)[source]#: Bases: nemo_gym.config_types.ResponsesAPIAgentServerTypeConfig, nemo_gym.config_types.BaseServerInstanceConfig

nemo_gym.config_types.ServerInstanceConfig#: None

nemo_gym.config_types.ServerInstanceConfigTypeAdapter#: ‘TypeAdapter(…)’

nemo_gym.config_types.maybe_get_server_instance_config( name: str, server_type_config_dict: Any, ) → Tuple[Optional[nemo_gym.config_types.ServerInstanceConfig], Optional[pydantic.ValidationError]][source]#: Returns ServerInstanceConfig if a valid server, otherwise None with error details

nemo_gym.config_types.is_almost_server(server_type_config_dict: Any) → bool[source]#: Detects if a config looks like a server but might fail validation.

nemo_gym.config_types.AGENT_REF_KEY#: ‘agent_ref’

nemo_gym.config_types#

Module Contents#

Classes#

Functions#

Data#

API#

`nemo_gym.config_types`#