stages.synthetic.qa_multilingual_synthetic

This module contains a simple stage for generating synthetic data. It takes in Empty task and a prompt and produces the output in form of a DocumentBatch.

Module Contents

Classes

QAMultilingualSyntheticStage

A simple stage for generating synthetic data. It takes in Empty task and a prompt and produces the output in form of a DocumentBatch.

API

class stages.synthetic.qa_multilingual_synthetic.QAMultilingualSyntheticStage

Bases: nemo_curator.stages.base.ProcessingStage[nemo_curator.tasks._EmptyTask, nemo_curator.tasks.DocumentBatch]

A simple stage for generating synthetic data. It takes in Empty task and a prompt and produces the output in form of a DocumentBatch.

client: nemo_curator.models.client.llm_client.AsyncLLMClient | nemo_curator.models.client.llm_client.LLMClient

None

generation_config: nemo_curator.models.client.llm_client.GenerationConfig | None

None

inputs() → tuple[list[str], list[str]]

Define stage input requirements.

Returns (tuple[list[str], list[str]]): Tuple of (required_attributes, required_columns) where: - required_top_level_attributes: List of task attributes that must be present - required_data_attributes: List of attributes within the data that must be present

languages: list[str]

None

model_name: str

None

name: str

‘QAMultilingualSyntheticStage’

num_samples: int

None

outputs() → tuple[list[str], list[str]]

Define stage output specification.

Returns (tuple[list[str], list[str]]): Tuple of (output_attributes, output_columns) where: - output_top_level_attributes: List of task attributes this stage adds/modifies - output_data_attributes: List of attributes within the data that this stage adds/modifies

process(

_: nemo_curator.tasks._EmptyTask,

) → nemo_curator.tasks.DocumentBatch

Process a task and return the result. Args: task (X): Input task to process Returns (Y | list[Y]): - Single task: For 1-to-1 transformations - List of tasks: For 1-to-many transformations (e.g., readers) - None: If the task should be filtered out

prompt: str

None

setup(

_: nemo_curator.backends.base.WorkerMetadata | None = None,

) → None

Setup method called once before processing begins. Override this method to perform any initialization that should happen once per worker. Args: worker_metadata (WorkerMetadata, optional): Information about the worker (provided by some backends)