Generate Data#
Generate synthetic text data using large language models (LLMs) for pre-training, fine-tuning, and evaluation tasks. Create high-quality training data for low-resource languages and domains, or perform knowledge distillation from existing models.
How it Works#
NeMo Curator’s synthetic data generation capabilities are organized into several components:
Model Integration: Connect to OpenAI-compatible model endpoints or self-hosted models
Generation Pipelines: Use pre-built pipelines for common generation tasks
Custom Workflows: Combine components to create specialized generation pipelines
Quality Control: Filter and validate generated data using NeMo Curator’s processing tools
Service Connections#
Connect your data generation workflows to powerful language models and scoring services. Choose from cloud-based APIs or deploy models in your own infrastructure.
Connect to OpenAI’s API endpoints for GPT models and other services
Deploy and connect to models using NVIDIA NeMo Deploy
Integrate reward models for quality scoring and filtering
Generation Pipelines#
Transform your data needs into production-ready synthetic datasets using specialized generation pipelines.
Q&A Generation Pipelines#
Use these pipelines to generate question-and-answer data for training, evaluation, and comprehension tasks.
Generate closed-ended questions about a given document. Ideal for creating evaluation or comprehension datasets.
Generate open-ended questions (“openlines”) for dialogue data, including macro topics, subtopics, and detailed revisions.
Generate diverse question-answer pairs from documents for QA datasets.
Content Transformation & Summarization#
Transform, rewrite, and summarize documents to create clear, concise, and structured text data.
Rewrite documents into a style similar to Wikipedia, improving clarity and scholarly tone.
Distill documents to concise summaries, removing redundancy and focusing on key information.
Extract key knowledge and facts from documents for summarization and analysis.
Extract structured knowledge lists from documents for downstream use.
Dialogue & Writing#
Create synthetic dialogues and writing tasks to support conversational and creative data generation.
Generate multi-turn dialogues and two-turn prompts for preference data. Synthesize conversations where an LLM plays both user and assistant.
Generate writing prompts (essays, poems, etc.) and revise them for detail and diversity. Useful for creative and instructional datasets.
STEM & Coding#
Generate math and coding problems, as well as classify entities for STEM-related datasets.
Generate math questions for dialogue data, including macro topics, subtopics, and problems at various school levels.
Generate Python coding problems for dialogue data, including macro topics, subtopics, and problems for various skill levels.
Classify entities (for example, Wikipedia entries) as math- or Python-related using an LLM. Useful for filtering or labeling data for downstream tasks.
Infrastructure & Customization#
Leverage asynchronous pipelines and customizable prompts to scale and tailor your data generation workflows.
Generate synthetic data in parallel using asynchronous pipelines for maximum efficiency. Ideal for large-scale prompt generation and working with rate-limited LLM APIs. Provides async alternatives to all major text data generation pipelines in NeMo Curator.
Integrations#
Combine generation with powerful filtering and processing capabilities.
Combine synthetic data generation with other NeMo Curator modules for filtering and processing