For AI agents: a documentation index is available at the root level at /llms.txt. Append /llms.txt to any URL for a page-level index, or .md for the markdown version of any page.
LogoLogoNeMo Curator
DocumentationAPI Reference
DocumentationAPI Reference
  • Home
    • Welcome
  • About NeMo Curator
    • Overview
    • Key Features
      • Overview
      • Deduplication
  • Get Started
    • Overview
    • Text Quickstart
    • Image Quickstart
    • Video Quickstart
    • Audio Quickstart
  • Curate Text
    • Overview
    • Tutorials
  • Curate Images
    • Overview
    • Save and Export
  • Curate Video
    • Overview
    • Load Data
    • Save and Export
  • Curate Audio
    • Overview
    • Save and Export
  • Setup & Deployment
    • Overview
    • Installation
  • Reference
    • Overview
    • Related Tools
  • Welcome
  • Overview
  • Key Features
  • Overview
  • Deduplication
  • Overview
  • Loading
  • Acquisition
  • Processing
  • Curation Pipeline
  • Overview
  • Loading
  • Data Processing
  • Data Export
  • Overview
  • Architecture
  • Abstractions
  • Data Flow
  • Overview
  • Curation Pipeline
  • Audio Batch
  • ASR Pipeline
  • Quality Metrics
  • Manifests and Ingest
  • Text Integration
  • Overview
  • Migration FAQ
  • Migration Guide
  • Overview
  • Text Quickstart
  • Image Quickstart
  • Video Quickstart
  • Audio Quickstart
  • Overview
  • Tutorials
  • Overview
  • ArXiv
  • Common Crawl
  • Custom Sources
  • Read Existing Data
  • Wikipedia
  • Overview
  • Overview
  • Add IDs
  • Text Cleaning
  • Overview
  • Exact Deduplication
  • Fuzzy Deduplication
  • Semantic Deduplication
  • Overview
  • Language Detection
  • Stopwords
  • Overview
  • Classifier
  • Distributed Classifier
  • Heuristic Filtering
  • Overview
  • Code Processing
  • Overview
  • Overview
  • Beginner Tutorial
  • Deduplication Workflow
  • Overview
  • TAR Archives
  • Overview
  • Overview
  • CLIP Embedder
  • Overview
  • Aesthetic Filter
  • NSFW Filter
  • Save and Export
  • Overview
  • Overview
  • Beginner Tutorial
  • Split and Dedup
  • Overview
  • Add Custom Environment
  • Add Custom Code
  • Add Custom Model
  • Add Custom Stage
  • Load Data
  • Overview
  • Clipping
  • Transcoding
  • Filtering
  • Embeddings
  • Deduplication
  • Frame Extraction
  • Captions Preview
  • Save and Export
  • Overview
  • Overview
  • Beginner Tutorial
  • Overview
  • Custom Manifests
  • FLEURS Dataset
  • Local Files
  • Overview
  • Overview
  • NeMo ASR Models
  • Overview
  • WER Filtering
  • Duration Filtering
  • Overview
  • Duration Calculation
  • Format Validation
  • Text Integration
  • Save and Export
  • Overview
  • Installation
  • Overview
  • Requirements
  • Overview
  • Overview
  • Overview
  • Memory Management
  • GPU Processing
  • Resumable Processing
  • Execution Backends
  • Container Environments
  • Related Tools
On this page
  • Modality Concepts
  • Universal Concepts
About NeMo CuratorConcepts

Concepts

||View as Markdown|

Learn about the core components and concepts introduced by NeMo Curator.

Modality Concepts

Learn about working with specific modalities using NeMo Curator.

Text Curation Concepts

Learn about text data curation, covering data loading and processing (filtering, classification, deduplication).

Image Curation Concepts

Explore key concepts for image data curation, including scalable loading, processing (embedding, classification, filtering), and dataset export.

Video Curation Concepts

Discover video data curation concepts, such as distributed processing, pipeline stages, execution modes, and efficient data flow.

Audio Curation Concepts

Learn about speech data curation, ASR inference, quality assessment, and audio-text integration workflows.

Universal Concepts

Core concepts that apply across all modalities in NeMo Curator.

Deduplication Concepts

Comprehensive overview of deduplication techniques across text, image, and video modalities including exact, fuzzy, and semantic approaches.

Previous

Key Features

Next

Deduplication Concepts

NVIDIANVIDIA
Developer-friendly docs for your API
Privacy Policy | Your Privacy Choices | Terms of Service | Accessibility | Corporate Policies | Product Security | Contact

Copyright © 2026, NVIDIA Corporation.