> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/curator/v26.04/about/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/curator/_mcp/server.

# About NeMo Curator

## Docs

- [Key Features](https://docs.nvidia.com/nemo/curator/v26.04/about/key-features.md): Comprehensive overview of NeMo Curator's key features for text, image, video, and audio data curation with deployment options
- [Overview](https://docs.nvidia.com/nemo/curator/v26.04/about/concepts.md): Core concepts and terminology for NeMo Curator across text, image, video, and audio data curation modalities
- [Deduplication](https://docs.nvidia.com/nemo/curator/v26.04/about/concepts/deduplication.md): Comprehensive overview of deduplication techniques across text, image, and video modalities including exact, fuzzy, and semantic approaches
- [Resource Allocation](https://docs.nvidia.com/nemo/curator/v26.04/about/concepts/scaling/resource-allocation.md): How NeMo Curator allocates CPUs, GPUs, and memory across pipeline stages
- [Streaming](https://docs.nvidia.com/nemo/curator/v26.04/about/concepts/scaling/streaming.md): How NeMo Curator's streaming architecture processes data in constant memory
- [Auto-Balancing](https://docs.nvidia.com/nemo/curator/v26.04/about/concepts/scaling/auto-balancing.md): How NeMo Curator automatically balances resources across pipeline stages
- [Throughput](https://docs.nvidia.com/nemo/curator/v26.04/about/concepts/scaling/throughput.md): How to scale NeMo Curator from a single GPU to multi-node clusters for maximum throughput
- [Overview](https://docs.nvidia.com/nemo/curator/v26.04/about/concepts/text.md): Essential concepts for text data curation including loading and processing.
- [Loading](https://docs.nvidia.com/nemo/curator/v26.04/about/concepts/text/data/loading.md): Core concepts for loading and managing text datasets using pipeline-based readers and DocumentBatch tasks
- [Acquisition](https://docs.nvidia.com/nemo/curator/v26.04/about/concepts/text/data/acquisition.md): Core concepts for acquiring text data from remote sources including DocumentDownloader, DocumentIterator, and DocumentExtractor components
- [Processing](https://docs.nvidia.com/nemo/curator/v26.04/about/concepts/text/data/processing.md): Text processing workflows including quality filtering, fuzzy deduplication, content cleaning, and pipeline design
- [Curation Pipeline](https://docs.nvidia.com/nemo/curator/v26.04/about/concepts/text/data/data-curation-pipeline.md): Comprehensive overview of NeMo Curator's text curation pipeline architecture including data acquisition and processing
- [Overview](https://docs.nvidia.com/nemo/curator/v26.04/about/concepts/image.md): Essential concepts for image data curation including loading, processing, and export with GPU acceleration
- [Loading](https://docs.nvidia.com/nemo/curator/v26.04/about/concepts/image/data/loading.md): Core concepts for loading and managing image datasets from tar archives with cloud storage support
- [Data Processing](https://docs.nvidia.com/nemo/curator/v26.04/about/concepts/image/data/processing.md): Core concepts for processing image data including embedding generation, filtering, and deduplication
- [Data Export](https://docs.nvidia.com/nemo/curator/v26.04/about/concepts/image/data/export.md): Core concepts for saving and exporting curated image datasets including metadata and resharding
- [Overview](https://docs.nvidia.com/nemo/curator/v26.04/about/concepts/video.md): Essential concepts for video data curation including distributed processing, pipeline stages, and execution modes
- [Architecture](https://docs.nvidia.com/nemo/curator/v26.04/about/concepts/video/architecture.md): Architecture overview of NeMo Curator's Ray-based distributed video curation system with autoscaling
- [Abstractions](https://docs.nvidia.com/nemo/curator/v26.04/about/concepts/video/abstractions.md): Key abstractions in video curation including stages, pipelines, and execution modes for scalable processing
- [Data Flow](https://docs.nvidia.com/nemo/curator/v26.04/about/concepts/video/data-flow.md): Understanding data flow in video curation pipelines including Ray object store and streaming optimization
- [Overview](https://docs.nvidia.com/nemo/curator/v26.04/about/concepts/audio.md): Essential concepts for audio data curation including ASR inference, quality assessment, and speech processing workflows
- [Curation Pipeline](https://docs.nvidia.com/nemo/curator/v26.04/about/concepts/audio/curation-pipeline.md): Modality-level overview of ingest, validation, optional ASR, metrics, filtering, and export
- [Audio Task](https://docs.nvidia.com/nemo/curator/v26.04/about/concepts/audio/audio-task.md): Understanding the AudioTask data structure for audio file management and validation in NeMo Curator
- [ASR Pipeline](https://docs.nvidia.com/nemo/curator/v26.04/about/concepts/audio/asr-pipeline.md): Comprehensive overview of the automatic speech recognition pipeline architecture and workflow in NeMo Curator
- [Quality Metrics](https://docs.nvidia.com/nemo/curator/v26.04/about/concepts/audio/quality-metrics.md): Core concepts for evaluating speech transcription quality using WER, CER, duration analysis, and speech rate metrics
- [Manifests and Ingest](https://docs.nvidia.com/nemo/curator/v26.04/about/concepts/audio/manifests-ingest.md): Concepts for constructing manifests and ingesting audio datasets in NeMo Curator
- [ALM Pipeline](https://docs.nvidia.com/nemo/curator/v26.04/about/concepts/audio/alm-pipeline.md): Audio Language Model (ALM) data curation pipeline for creating training windows from diarized audio segments
- [Text Integration](https://docs.nvidia.com/nemo/curator/v26.04/about/concepts/audio/text-integration.md): Concepts for integrating audio processing with text curation workflows in multimodal applications
- [Overview](https://docs.nvidia.com/nemo/curator/v26.04/about/release-notes.md): Release notes and version history for NeMo Curator platform updates and new features
- [Migration FAQ](https://docs.nvidia.com/nemo/curator/v26.04/about/release-notes/migration-faq.md): Frequently asked questions about migrating NeMo Curator workflows from Dask to Ray
- [Migration Guide](https://docs.nvidia.com/nemo/curator/v26.04/about/release-notes/migration-guide.md): Guide for migrating NeMo Curator workflows from Dask to Ray-based pipeline architecture


> **Note:** This page contains both a page directory (above) and the landing page content (below). The page directory is generated for agent use and does not appear on the landing page.

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/curator/v26.04/about/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/curator/_mcp/server.

> Overview of NeMo Curator, an open-source platform for scalable data curation across text, image, video, and audio modalities for AI training

# Overview of NeMo Curator

NeMo Curator is an open-source, enterprise-grade platform for scalable, privacy-aware data curation across text, image, video, and audio modalities.

NeMo Curator, part of the NVIDIA NeMo software suite for managing the AI agent lifecycle, helps you prepare high-quality, compliant datasets for large language model (LLM) and generative artificial intelligence (AI) training. Whether you work in the cloud, on-premises, or in a hybrid environment, NeMo Curator supports your workflow.

## Target Users

* **Data scientists and machine learning engineers**: Build and curate datasets for LLMs, generative models, and multimodal AI.

* **Cluster administrators and DevOps professionals**: Deploy and scale curation pipelines.

* **Researchers**: Experiment with new data curation techniques and ablation studies.

* **Enterprises**: Ensure data privacy, compliance, and quality for production AI workflows.

## How It Works

NeMo Curator speeds up data curation by using modern hardware and distributed computing frameworks. You can process data efficiently—from a single laptop to a multi-node GPU cluster. With modular pipelines, advanced filtering, and easy integration with machine learning operations (MLOps) tools, leading organizations trust NeMo Curator.

* **Text Curation**: Uses a pipeline-based architecture with modular processing stages running on Ray. Data flows through data download, extraction, language detection, rule-based quality filtering, deduplication (exact, fuzzy and semantic) and model based quality filtering.
* **Image Curation**: Uses pipeline-based architecture with modular stages for loading, embedding generation, classification (aesthetic, NSFW), filtering, and export workflows. Supports distributed processing with optional GPU acceleration.
* **Video Curation**: Employs Ray-based pipelines to split long videos into clips using fixed stride or scene-change detection, with optional encoding, filtering, embedding generation, and deduplication for large-scale video processing.
* **Audio Curation**: Provides ASR inference using models, quality assessment through Word Error Rate (WER) calculation, duration analysis, and integration with text curation workflows for speech data processing.

### Key Technologies

* **Graphics Processing Units (GPUs)**: Speed up data processing for large-scale workloads.
* **Distributed Computing**: Uses Ray and NVIDIA RAPIDS libraries for scalable, parallel processing.
* **Modular Pipelines**: Build, customize, and scale curation workflows to fit your needs.
* **MLOps Integration**: Seamlessly connects with modern MLOps environments for production-ready workflows.

## Concepts

Explore the foundational concepts and terminology used across NeMo Curator.

#### [Text Curation Concepts](/about/concepts/text)

Learn about text data curation, covering data loading and processing (filtering, deduplication, classification).

#### [Image Curation Concepts](/about/concepts/image)

Explore key concepts for image data curation, including scalable loading, processing (embedding, classification, filtering, deduplication), and dataset export.

#### [Video Curation Concepts](/about/concepts/video)

Discover video data curation concepts, such as distributed processing, pipeline stages, execution modes, and efficient data flow.

#### [Audio Curation Concepts](/about/concepts/audio)

Learn about speech data curation, ASR inference, quality assessment, and audio-text integration workflows.