nemo_curator.backends.failed_task_markers

Module Contents

Functions

Name	Description
`_configure_failed_task_manifest_dir`	-
`configure_failed_task_manifest_dir`	Configure a local attempt-scoped FailedTask manifest directory unless overridden.
`configure_slurm_array_failed_task_manifest_dir`	Configure an attempt-scoped FailedTask manifest directory unless overridden.
`failed_task_manifest_exists`	Return whether the current attempt has recorded any FailedTask.
`record_failed_tasks`	Write one attempt-scoped manifest after any FailedTask is detected.

Data

FAILED_TASKS_DIR_ENV_VAR

FAILED_TASK_MANIFEST_FILENAME

API

nemo_curator.backends.failed_task_markers._configure_failed_task_manifest_dir(
    default_dir: pathlib.Path
) -> pathlib.Path

nemo_curator.backends.failed_task_markers.configure_failed_task_manifest_dir(
    checkpoint_path: str | pathlib.Path
) -> pathlib.Path

Configure a local attempt-scoped FailedTask manifest directory unless overridden.

nemo_curator.backends.failed_task_markers.configure_slurm_array_failed_task_manifest_dir(
    checkpoint_path: str | pathlib.Path,
    shard_index: int
) -> pathlib.Path

Configure an attempt-scoped FailedTask manifest directory unless overridden.

nemo_curator.backends.failed_task_markers.failed_task_manifest_exists(
    manifest_dir: str | pathlib.Path | None = None
) -> bool

Return whether the current attempt has recorded any FailedTask.

nemo_curator.backends.failed_task_markers.record_failed_tasks() -> None

Write one attempt-scoped manifest after any FailedTask is detected.

nemo_curator.backends.failed_task_markers.FAILED_TASKS_DIR_ENV_VAR = 'NEMO_CURATOR_FAILED_TASKS_DIR'

nemo_curator.backends.failed_task_markers.FAILED_TASK_MANIFEST_FILENAME = 'failed_tasks.json'