nemo_curator.stages.interleaved.utils.validation_utils

View as Markdown

Module Contents

Functions

NameDescription
require_source_id_field-
resolve_storage_options-
validate_and_project_source_fieldsValidate requested source fields and normalize selected values for tabular output.

API

nemo_curator.stages.interleaved.utils.validation_utils.require_source_id_field(
source_id_field: str
) -> str
nemo_curator.stages.interleaved.utils.validation_utils.resolve_storage_options(
task: nemo_curator.tasks.Task[typing.Any] | None = None,
io_kwargs: dict[str, object] | None = None
) -> dict[str, object]
nemo_curator.stages.interleaved.utils.validation_utils.validate_and_project_source_fields(
sample: dict[str, typing.Any],
fields: tuple[str, ...] | None,
excluded_fields: set[str]
) -> dict[str, typing.Any]

Validate requested source fields and normalize selected values for tabular output.