> For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/nemo/curator/v26.09/about/concepts/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/curator/_mcp/server. # Concepts ## Docs - [Deduplication](https://docs.nvidia.com/nemo/curator/v26.09/about/concepts/deduplication.md): Comprehensive overview of deduplication techniques across text, image, and video modalities including exact, fuzzy, and semantic approaches - [Resource Allocation](https://docs.nvidia.com/nemo/curator/v26.09/about/concepts/scaling/resource-allocation.md): How NeMo Curator allocates CPUs, GPUs, and memory across pipeline stages - [Streaming](https://docs.nvidia.com/nemo/curator/v26.09/about/concepts/scaling/streaming.md): How NeMo Curator's streaming architecture processes data in constant memory - [Auto-Balancing](https://docs.nvidia.com/nemo/curator/v26.09/about/concepts/scaling/auto-balancing.md): How NeMo Curator automatically balances resources across pipeline stages - [Throughput](https://docs.nvidia.com/nemo/curator/v26.09/about/concepts/scaling/throughput.md): How to scale NeMo Curator from a single GPU to multi-node clusters for maximum throughput - [Overview](https://docs.nvidia.com/nemo/curator/v26.09/about/concepts/text.md): Essential concepts for text data curation including loading and processing. - [Loading](https://docs.nvidia.com/nemo/curator/v26.09/about/concepts/text/data/loading.md): Core concepts for loading and managing text datasets using pipeline-based readers and DocumentBatch tasks - [Acquisition](https://docs.nvidia.com/nemo/curator/v26.09/about/concepts/text/data/acquisition.md): Core concepts for acquiring text data from remote sources including DocumentDownloader, DocumentIterator, and DocumentExtractor components - [Processing](https://docs.nvidia.com/nemo/curator/v26.09/about/concepts/text/data/processing.md): Text processing workflows including quality filtering, fuzzy deduplication, content cleaning, and pipeline design - [Curation Pipeline](https://docs.nvidia.com/nemo/curator/v26.09/about/concepts/text/data/data-curation-pipeline.md): Comprehensive overview of NeMo Curator's text curation pipeline architecture including data acquisition and processing - [Overview](https://docs.nvidia.com/nemo/curator/v26.09/about/concepts/image.md): Essential concepts for image data curation including loading, processing, and export with GPU acceleration - [Loading](https://docs.nvidia.com/nemo/curator/v26.09/about/concepts/image/data/loading.md): Core concepts for loading and managing image datasets from tar archives with cloud storage support - [Data Processing](https://docs.nvidia.com/nemo/curator/v26.09/about/concepts/image/data/processing.md): Core concepts for processing image data including embedding generation, filtering, and deduplication - [Data Export](https://docs.nvidia.com/nemo/curator/v26.09/about/concepts/image/data/export.md): Core concepts for saving and exporting curated image datasets including metadata and resharding - [Overview](https://docs.nvidia.com/nemo/curator/v26.09/about/concepts/video.md): Essential concepts for video data curation including distributed processing, pipeline stages, and execution modes - [Architecture](https://docs.nvidia.com/nemo/curator/v26.09/about/concepts/video/architecture.md): Architecture overview of NeMo Curator's Ray-based distributed video curation system with autoscaling - [Abstractions](https://docs.nvidia.com/nemo/curator/v26.09/about/concepts/video/abstractions.md): Key abstractions in video curation including stages, pipelines, and execution modes for scalable processing - [Data Flow](https://docs.nvidia.com/nemo/curator/v26.09/about/concepts/video/data-flow.md): Understanding data flow in video curation pipelines including Ray object store and streaming optimization - [Overview](https://docs.nvidia.com/nemo/curator/v26.09/about/concepts/audio.md): Essential concepts for audio data curation including ASR inference, quality assessment, and speech processing workflows - [Curation Pipeline](https://docs.nvidia.com/nemo/curator/v26.09/about/concepts/audio/curation-pipeline.md): Modality-level overview of ingest, validation, optional ASR, metrics, filtering, and export - [Audio Task](https://docs.nvidia.com/nemo/curator/v26.09/about/concepts/audio/audio-task.md): Understanding the AudioTask data structure for audio file management and validation in NeMo Curator - [ASR Pipeline](https://docs.nvidia.com/nemo/curator/v26.09/about/concepts/audio/asr-pipeline.md): Comprehensive overview of the automatic speech recognition pipeline architecture and workflow in NeMo Curator - [Quality Metrics](https://docs.nvidia.com/nemo/curator/v26.09/about/concepts/audio/quality-metrics.md): Core concepts for evaluating speech transcription quality using WER, CER, duration analysis, and speech rate metrics - [Manifests and Ingest](https://docs.nvidia.com/nemo/curator/v26.09/about/concepts/audio/manifests-ingest.md): Concepts for constructing manifests and ingesting audio datasets in NeMo Curator - [ALM Pipeline](https://docs.nvidia.com/nemo/curator/v26.09/about/concepts/audio/alm-pipeline.md): Audio Language Model (ALM) data curation pipeline for creating training windows from diarized audio segments - [Text Integration](https://docs.nvidia.com/nemo/curator/v26.09/about/concepts/audio/text-integration.md): Concepts for integrating audio processing with text curation workflows in multimodal applications > **Note:** This page contains both a page directory (above) and the landing page content (below). The page directory is generated for agent use and does not appear on the landing page. > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/nemo/curator/v26.09/about/concepts/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/curator/_mcp/server. > Core concepts and terminology for NeMo Curator across text, image, video, and audio data curation modalities # Concepts Learn about the core components and concepts introduced by NeMo Curator. ## Modality Concepts Learn about working with specific modalities using NeMo Curator. #### [Text Curation Concepts](/about/concepts/text) Learn about text data curation, covering data loading and processing (filtering, classification, deduplication). #### [Image Curation Concepts](/about/concepts/image) Explore key concepts for image data curation, including scalable loading, processing (embedding, classification, filtering), and dataset export. #### [Video Curation Concepts](/about/concepts/video) Discover video data curation concepts, such as distributed processing, pipeline stages, execution modes, and efficient data flow. #### [Audio Curation Concepts](/about/concepts/audio) Learn about speech data curation, ASR inference, quality assessment, and audio-text integration workflows. ## Universal Concepts Core concepts that apply across all modalities in NeMo Curator. #### [Deduplication](/about/concepts/deduplication) Comprehensive overview of deduplication techniques across text, image, and video modalities including exact, fuzzy, and semantic approaches. #### [Resource Allocation](/about/concepts/scaling/resource-allocation) How NeMo Curator allocates CPUs, GPUs, and memory across pipeline stages for optimal hardware utilization. #### [Streaming](/about/concepts/scaling/streaming) How streaming execution processes data in batches for constant memory usage and higher GPU utilization. #### [Auto-Balancing](/about/concepts/scaling/auto-balancing) How the executor automatically balances resources across pipeline stages to eliminate bottlenecks. #### [Throughput](/about/concepts/scaling/throughput) Scale from a single GPU to multi-node clusters with near-linear scaling.