Nemotron-CC Pipelines

Nemotron-CC provides advanced synthetic data generation workflows for transforming and extracting knowledge from existing text documents. Unlike simple generation, these pipelines use sophisticated preprocessing, LLM-based transformation, and postprocessing to create high-quality training data.

The Composable Pipeline Pattern

Nemotron-CC stages follow a composable pattern with three distinct phases:

Preprocessing: Segment documents, filter by length, and prepare inputs for the LLM
Generation: Apply task-specific prompts to transform text using the LLM
Postprocessing: Clean outputs, remove formatting artifacts, and filter low-quality results

This separation enables fine-grained control over each phase while providing reusable helper functions for common patterns.

Pipeline Architecture

Input Data Requirements

Before running a Nemotron-CC pipeline, prepare your input data as Parquet files with the required schema.

Required Schema

Column	Type	Description
`id`	`int64`	Unique document identifier. Required by the preprocessing pipeline to reassemble document segments after splitting.
`text`	`string`	Document content to transform. This is the primary input field for all Nemotron-CC stages.
`bucketed_results`	`int64`	Quality score used to route documents to appropriate pipelines. Values typically range from 0-20, where higher scores indicate higher quality content.

Quality Score Field

The bucketed_results field contains quality scores that determine which pipeline processes each document:

High-quality documents (bucketed_results >11): Process with DiverseQA, Distill, ExtractKnowledge, or KnowledgeList tasks
Low-quality documents (bucketed_results <= 11): Process with WikipediaParaphrasing to improve text quality

Generating Quality Scores

Use NeMo Curator’s quality assessment tools to generate quality scores before running SDG pipelines:

1 from nemo_curator.pipeline import Pipeline
2 from nemo_curator.stages.text.io.reader import JsonlReader
3 from nemo_curator.stages.text.io.writer import ParquetWriter
4 from nemo_curator.stages.text.classifiers import FineWebEduClassifier
5 from nemo_curator.stages.text.modules import AddId
6 
7 # Create pipeline to score documents
8 pipeline = Pipeline(name="quality_scoring")
9 
10 # Read raw documents
11 pipeline.add_stage(JsonlReader(file_paths="raw_data/*.jsonl", fields=["text"]))
12 
13 # Add unique document IDs
14 pipeline.add_stage(AddId(id_field="id"))
15 
16 # Score document quality (outputs int score 0-5)
17 pipeline.add_stage(
18     FineWebEduClassifier(
19         int_score_field="bucketed_results",  # Use this as quality score
20     )
21 )
22 
23 # Save as Parquet for SDG pipeline
24 pipeline.add_stage(ParquetWriter(path="scored_data/"))
25 
26 results = pipeline.run()

The example above uses FineWebEduClassifier which outputs scores 0-5. For the Nemotron-CC threshold of 11, you can either:

Scale the scores (e.g., multiply by 4)
Adjust the filter threshold in your SDG pipeline
Use a different classifier that outputs scores in the 0-20 range

For detailed information on quality scoring options, see Quality Assessment & Filtering .

Example Data

An example Parquet file with the correct schema is available in the tutorials directory:

$ tutorials/synthetic/nemotron_cc/example_data/data.parquet

You can inspect its structure:

1 import pandas as pd
2 
3 df = pd.read_parquet("tutorials/synthetic/nemotron_cc/example_data/data.parquet")
4 print(df.columns.tolist())  # ['id', 'text', 'bucketed_results']
5 print(df.head(2))

Available Tasks

Nemotron-CC provides five specialized generation tasks, each designed for specific data transformation needs:

Task	Stage Class	Purpose	Use Case
Wikipedia Paraphrasing	`WikipediaParaphrasingStage`	Rewrite text as Wikipedia-style prose	Improving noisy web data
Diverse QA	`DiverseQAStage`	Generate diverse Q&A pairs	Reading comprehension training
Distill	`DistillStage`	Create condensed, informative paraphrases	Knowledge distillation
Extract Knowledge	`ExtractKnowledgeStage`	Extract factual content as passages	Knowledge base creation
Knowledge List	`KnowledgeListStage`	Extract structured fact lists	Fact extraction

Quality-Based Processing Strategy

Nemotron-CC pipelines are designed to process data based on quality scores. The typical approach:

High-Quality Data Pipeline

For documents with high quality scores, use tasks that leverage the existing quality:

DiverseQA: Generate Q&A pairs from well-structured content
Distill: Create condensed versions preserving key information
ExtractKnowledge: Extract factual passages
KnowledgeList: Extract structured facts

1 from nemo_curator.stages.text.filters import Filter
2 
3 # Filter for high-quality documents (score &gt;11)
4 pipeline.add_stage(
5     Filter(
6         filter_fn=lambda x: int(x) &gt;11,
7         filter_field="bucketed_results",
8     ),
9 )

Low-Quality Data Pipeline

For documents with lower quality scores, use Wikipedia Paraphrasing to improve text quality:

1 # Filter for low-quality documents (score &lt;= 11)
2 pipeline.add_stage(
3     Filter(
4         filter_fn=lambda x: int(x) &lt;= 11,
5         filter_field="bucketed_results",
6     ),
7 )

Using Helper Functions

The recommended approach is to use the helper functions in nemotron_cc_pipelines.py:

The nemotron_cc_pipelines helper functions are provided in the tutorials directory, not as part of the installed package. Copy the nemotron_cc_pipelines.py file to your project or reference the patterns when building custom pipelines.

1 from nemotron_cc_pipelines import (
2     add_preprocessing_pipeline,
3     add_diverse_qa_postprocessing_pipeline,
4 )
5 from nemo_curator.pipeline import Pipeline
6 from nemo_curator.stages.synthetic.nemotron_cc.nemotron_cc import DiverseQAStage
7 
8 pipeline = Pipeline(name="diverse_qa_pipeline")
9 
10 # Add preprocessing
11 pipeline = add_preprocessing_pipeline(
12     pipeline=pipeline,
13     text_field="text",
14     system_prompt=SYSTEM_PROMPT,
15     user_prompt_template=PROMPT_TEMPLATE,
16     min_document_tokens=30,
17     min_segment_tokens=30,
18     max_input_tokens=1000,
19     args=args,  # Contains tokenizer config
20 )
21 
22 # Add generation stage
23 pipeline.add_stage(
24     DiverseQAStage(
25         client=llm_client,
26         model_name="meta/llama-3.3-70b-instruct",
27         generation_config=generation_config,
28         input_field="text",
29         output_field="diverse_qa",
30     )
31 )
32 
33 # Add postprocessing
34 pipeline = add_diverse_qa_postprocessing_pipeline(
35     pipeline=pipeline,
36     llm_response_field="diverse_qa",
37     args=args,
38 )

Task Configuration

Each task has specific token count and preprocessing requirements:

Task	Min Doc Tokens	Min Segment Tokens	Max Input Tokens	Max Output Tokens
Diverse QA	30	30	1000	600
Distill	30	10	2000	1600
Extract Knowledge	30	30	1400	1400
Knowledge List	30	30	1000	600
Wikipedia Paraphrasing	5	5	512	512

Quick Example

1 import os
2 from transformers import AutoTokenizer
3 from nemo_curator.core.client import RayClient
4 from nemo_curator.backends.xenna import XennaExecutor
5 from nemo_curator.models.client import AsyncOpenAIClient
6 from nemo_curator.models.client.llm_client import GenerationConfig
7 from nemo_curator.pipeline import Pipeline
8 from nemo_curator.stages.synthetic.nemotron_cc.nemotron_cc import DiverseQAStage
9 from nemo_curator.stages.text.io.reader.parquet import ParquetReader
10 from nemo_curator.stages.text.io.writer.parquet import ParquetWriter
11 
12 # Initialize
13 client = RayClient(include_dashboard=False)
14 client.start()
15 tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.3-70B-Instruct")
16 
17 # Create LLM client
18 llm_client = AsyncOpenAIClient(
19     api_key=os.environ["NVIDIA_API_KEY"],
20     base_url="https://integrate.api.nvidia.com/v1",
21     max_concurrent_requests=5,
22 )
23 
24 # Build pipeline (see "Using Helper Functions" section for preprocessing/postprocessing)
25 pipeline = Pipeline(name="nemotron_cc_diverse_qa")
26 pipeline.add_stage(ParquetReader(file_paths=["./input_data/*.parquet"]))
27 
28 # Add preprocessing stages using helper function:
29 # pipeline = add_preprocessing_pipeline(pipeline, text_field="text", ...)
30 
31 # Add generation stage
32 pipeline.add_stage(
33     DiverseQAStage(
34         client=llm_client,
35         model_name="meta/llama-3.3-70b-instruct",
36         generation_config=GenerationConfig(temperature=0.5, top_p=0.9),
37         input_field="text",
38         output_field="diverse_qa",
39     )
40 )
41 
42 # Add postprocessing stages using helper function:
43 # pipeline = add_diverse_qa_postprocessing_pipeline(pipeline, llm_response_field="diverse_qa", ...)
44 
45 pipeline.add_stage(ParquetWriter(path="./output/"))
46 
47 # Execute
48 executor = XennaExecutor()
49 results = pipeline.run(executor)
50 
51 client.stop()

NDD-Backed Stages

All five Nemotron-CC tasks have NDD-backed equivalents that replace the AsyncOpenAIClient with NeMo Data Designer execution. These stages share the same input_field, output_field, and prompt interface, but configure the LLM through NDD’s ModelConfig and ModelProvider instead of an AsyncOpenAIClient.

Import the NDD-backed stages from nemo_curator.stages.synthetic.nemotron_cc.nemo_data_designer.nemotron_cc:

1 import os
2 
3 import data_designer.config as dd
4 
5 from nemo_curator.pipeline import Pipeline
6 from nemo_curator.stages.synthetic.nemotron_cc.nemo_data_designer.nemotron_cc import DiverseQAStage
7 
8 model_config = dd.ModelConfig(
9     alias="meta/llama-3.3-70b-instruct",
10     model="meta/llama-3.3-70b-instruct",
11     provider="nvidia",
12     inference_parameters=dd.ChatCompletionInferenceParams(
13         temperature=0.5, top_p=0.9, max_tokens=600,
14     ),
15 )
16 
17 model_provider = dd.ModelProvider(
18     name="nvidia",
19     endpoint="https://integrate.api.nvidia.com/v1",
20     provider_type="openai",
21     api_key=os.environ["NVIDIA_API_KEY"],
22 )
23 
24 pipeline = Pipeline(name="nemotron_cc_ndd_diverse_qa")
25 pipeline.add_stage(
26     DiverseQAStage(
27         input_field="text",
28         output_field="diverse_qa",
29         model_alias="meta/llama-3.3-70b-instruct",
30         model_configs=[model_config],
31         model_providers=[model_provider],
32     )
33 )

The NDD backend provides automatic token metric collection and supports both local InferenceServer and remote NVIDIA NIM endpoints. See the NeMo Data Designer guide for full configuration details.

Detailed Reference

Task Reference

Detailed reference for each Nemotron-CC stage, prompts, and post-processing reference api

NeMo Data Designer

Declarative generation with NDD-backed Nemotron-CC stages ndd declarative