Text Data Generation Pipelines#
NeMo Curator provides pre-built pipelines for generating high-quality synthetic text data. These pipelines implement proven approaches for creating training data across different formats and styles.
Q&A Generation Pipelines#
Use these pipelines to generate question-and-answer data for training, evaluation, and comprehension tasks.
Generate closed-ended questions about a given document. Ideal for creating evaluation or comprehension datasets.
Generate open-ended questions (“openlines”) for dialogue data, including macro topics, subtopics, and detailed revisions.
Generate diverse question-answer pairs from documents for QA datasets.
Content Transformation & Summarization#
Transform, rewrite, and summarize documents to create clear, concise, and structured text data.
Rewrite documents into a style similar to Wikipedia, improving clarity and scholarly tone.
Distill documents to concise summaries, removing redundancy and focusing on key information.
Extract key knowledge and facts from documents for summarization and analysis.
Extract structured knowledge lists from documents for downstream use.
Dialogue & Writing#
Create synthetic dialogues and writing tasks to support conversational and creative data generation.
Generate multi-turn dialogues and two-turn prompts for preference data. Synthesize conversations where an LLM plays both user and assistant.
Generate writing prompts (essays, poems, etc.) and revise them for detail and diversity. Useful for creative and instructional datasets.
STEM & Coding#
Generate math and coding problems, as well as classify entities for STEM-related datasets.
Generate math questions for dialogue data, including macro topics, subtopics, and problems at various school levels.
Generate Python coding problems for dialogue data, including macro topics, subtopics, and problems for various skill levels.
Classify entities (for example, Wikipedia entries) as math- or Python-related using an LLM. Useful for filtering or labeling data for downstream tasks.
Infrastructure & Customization#
Leverage asynchronous pipelines and customizable prompts to scale and tailor your data generation workflows.
Generate synthetic data in parallel using asynchronous pipelines for maximum efficiency. Ideal for large-scale prompt generation and working with rate-limited LLM APIs. Provides async alternatives to all major text data generation pipelines in NeMo Curator.
References#
Find additional resources and guidance for customizing prompts and using generation pipelines effectively.
Customize prompt templates for any generation step. Use built-in or user-defined templates to control LLM behavior.