nemo_curator.backends.utils

Module Contents

Functions

Name	Description
`_logger_custom_deserializer`	-
`_logger_custom_serializer`	-
`merge_executor_configs`	Recursively merge two executor configs with deep merging of nested dicts.
`register_loguru_serializer`	Initialize a new local Ray cluster or connects to an existing one.
`warn_on_env_var_override`	-

API

nemo_curator.backends.utils._logger_custom_deserializer(
    _: None
) -> loguru.Logger

nemo_curator.backends.utils._logger_custom_serializer(
    _: loguru.Logger
) -> None

nemo_curator.backends.utils.merge_executor_configs(
    base_config: dict | None,
    override_config: dict | None
) -> dict

Recursively merge two executor configs with deep merging of nested dicts.

Parameters:

base_config

dict | None

Base configuration dictionary

override_config

dict | None

Configuration to merge on top of base_config

Returns: dict

Merged configuration dictionary with all nested dicts recursively merged

Examples:

>>> base = {"runtime_env": {"env_vars": {"A": "1", "B": "2"}}}
>>> override = {"runtime_env": {"env_vars": {"B": "3", "C": "4"}}}
>>> merge_executor_configs(base, override)
{"runtime_env": {"env_vars": {"A": "1", "B": "3", "C": "4"}}}

nemo_curator.backends.utils.register_loguru_serializer() -> None

Initialize a new local Ray cluster or connects to an existing one.

nemo_curator.backends.utils.warn_on_env_var_override(
    existing_config: dict | None,
    merged_config: dict | None
) -> None