For AI agents: a documentation index is available at the root level at /llms.txt. Append /llms.txt to any URL for a page-level index, or .md for the markdown version of any page.
LogoLogoNeMo Curator
DocumentationAPI Reference
DocumentationAPI Reference
  • API Reference
    • Overview
        • Nemo Curator
          • Backends
            • Base
            • Failed Task Markers
            • Internal
            • Ray Actor Pool
            • Ray Data
            • Slurm Array
            • Utils
            • Xenna
          • Config
          • Core
          • Metrics
          • Models
          • Package Info
          • Pipeline
          • Stages
          • Tasks
          • Utils
    • Pipeline
    • ProcessingStage
    • CompositeStage
    • Resources
  • Overview
  • Nemo Curator
  • Backends
  • Base
  • Failed Task Markers
  • Internal
  • Raft
  • Ray Comms
  • Ray Actor Pool
  • Adapter
  • Executor
  • Raft Adapter
  • Shuffle Adapter
  • Utils
  • Ray Data
  • Adapter
  • Executor
  • Utils
  • Slurm Array
  • Utils
  • Xenna
  • Adapter
  • Executor
  • Config
  • Run
  • Core
  • Client
  • Constants
  • Serve
  • Base
  • Constants
  • Dynamo
  • Backend
  • Config
  • Constants
  • Infra
  • Vllm
  • Placement
  • Ray Serve
  • Backend
  • Config
  • Server
  • Subprocess Mgr
  • Utils
  • Metrics
  • Constants
  • Start Prometheus Grafana
  • Utils
  • Models
  • Aesthetics
  • Base
  • Client
  • Llm Client
  • Openai Client
  • Clip
  • Cosmos Embed1
  • Nemotron 3 Nano Omni
  • Nemotron H Vl
  • Nsfw
  • Omni
  • Base
  • Prompt Formatter
  • Qwen Lm
  • Qwen Vl
  • Transnetv2
  • Vllm Model
  • Package Info
  • Pipeline
  • Pipeline
  • Workflow
  • Stages
  • Audio
  • Advanced Pipelines
  • Audio Data Filter
  • Audio Data Filter
  • Config
  • Alm
  • Alm Data Builder
  • Alm Data Overlap
  • Pretrain
  • Extraction
  • Finalize
  • Io
  • Pipeline
  • Planning
  • Utils
  • Common
  • Datasets
  • File Utils
  • Fleurs
  • Create Initial Manifest
  • Readspeech
  • Create Initial Manifest
  • Filtering
  • Band
  • Band Filter Module
  • Features
  • Predict
  • Sigmos
  • Utmos
  • Inference
  • Asr
  • Asr Nemo
  • Speaker Diarization
  • Pyannote
  • Sortformer
  • Vad
  • Whisperx Vad
  • Io
  • Convert
  • Extract Segments
  • Metrics
  • Bandwidth
  • Squim
  • Wer
  • Postprocessing
  • Timestamp Mapper
  • Preprocessing
  • Concatenation
  • Mono Conversion
  • Segmentation
  • Speaker Separation
  • Speaker Separation Module
  • Speaker Sep
  • Vad Segmentation
  • Tagging
  • Inference
  • Nemo Asr Align
  • Merge Alignment Diarization
  • Prepare Module Segments
  • Resample Audio
  • Split
  • Text
  • Chinese Conversion
  • Itn
  • Utils
  • Base
  • Client Partitioning
  • Deduplication
  • Exact
  • Identification
  • Workflow
  • Fuzzy
  • Buckets To Edges
  • Connected Components
  • Identify Duplicates
  • Lsh
  • Lsh
  • Stage
  • Minhash
  • Utils
  • Workflow
  • Gpu Utils
  • Id Generator
  • Io Utils
  • Semantic
  • Identify Duplicates
  • Kmeans
  • Pairwise
  • Pairwise Io
  • Ranking
  • Utils
  • Workflow
  • Shuffle Utils
  • Rapidsmpf Shuffler
  • Stage
  • File Partitioning
  • Function Decorators
  • Image
  • Deduplication
  • Removal
  • Embedders
  • Clip Embedder
  • Filters
  • Aesthetic Filter
  • Base
  • Nsfw Filter
  • Io
  • Convert
  • Image Reader
  • Image Writer
  • Interleaved
  • Filter
  • Blur Filter
  • Clip Score Filter
  • Image To Text Ratio Filter
  • Qrcode Filter
  • Io
  • Reader
  • Readers
  • Base
  • Parquet
  • Webdataset
  • Writers
  • Base
  • Tabular
  • Webdataset
  • Lance
  • Reader
  • Pdf
  • Nemotron Parse
  • Composite
  • Inference
  • Partitioning
  • Postprocess
  • Preprocess
  • Utils
  • Stages
  • Utils
  • Constants
  • Image Utils
  • Materialization
  • Schema
  • Validation Utils
  • Math
  • Classifiers
  • Finemath
  • Download
  • Extract
  • Html Extractors
  • Lynx
  • Mime Types
  • Modifiers
  • Chunking
  • Llm Cleanup
  • Merge Chunks
  • Resources
  • Synthetic
  • Nemo Data Designer
  • Data Designer
  • Nemotron Cc
  • Base
  • Nemo Data Designer
  • Base
  • Nemotron Cc
  • Nemotron Cc
  • Prompts
  • Omni
  • Base
  • Io
  • Ocr Conversationalize
  • Ocr Dense Qa
  • Ocr Nemotron V2
  • Ocr Scoring Qa
  • Utils
  • Conversation
  • Qa Multilingual Synthetic
  • Text
  • Classifiers
  • Aegis
  • Aegis Utils
  • Base
  • Content Type
  • Domain
  • Fineweb Edu
  • Prompt Task Complexity
  • Quality
  • Utils
  • Deduplication
  • Removal
  • Removal Workflow
  • Semantic
  • Download
  • Arxiv
  • Download
  • Extract
  • Iterator
  • Stage
  • Url Generation
  • Base
  • Download
  • Extract
  • Iterator
  • Stage
  • Url Generation
  • Common Crawl
  • Download
  • Extract
  • Stage
  • Url Generation
  • Warc Iterator
  • Html Extractors
  • Base
  • Justext
  • Resiliparse
  • Trafilatura
  • Utils
  • Ja Stopwords
  • Th Stopwords
  • Zh Stopwords
  • Utils
  • Wikipedia
  • Constants
  • Download
  • Extract
  • Iterator
  • Stage
  • Url Generation
  • Embedders
  • Base
  • Utils
  • Vllm
  • Experimental
  • Translation
  • Backends
  • Aws
  • Base
  • Google
  • Nmt
  • Evaluation
  • Faith
  • Text Quality
  • Pipeline
  • Stages
  • Format Translation Output
  • Merge Faith Scores
  • Reassembly
  • Segmentation
  • Skipped Rows
  • Translate
  • Utils
  • Async Utils
  • Field Paths
  • Metadata
  • Prompt Loader
  • Filters
  • Doc Filter
  • Fasttext
  • Fasttext Filters
  • Heuristic
  • Code
  • Code
  • Repetition
  • Repetition
  • String
  • Histogram
  • Histogram
  • Score Filter
  • Token
  • Token Count
  • Io
  • Reader
  • Base
  • Jsonl
  • Lance
  • Parquet
  • Writer
  • Base
  • Jsonl
  • Lance
  • Megatron Tokenizer
  • Parquet
  • Utils
  • Models
  • Model
  • Tokenizer
  • Utils
  • Modifiers
  • Doc Modifier
  • Fasttext
  • Fasttext Label
  • Modifier
  • String
  • C4
  • Line Remover
  • Markdown Remover
  • Newline Normalizer
  • Quotation Remover
  • Slicer
  • Url Remover
  • Unicode
  • Unicode Reformatter
  • Modules
  • Add Id
  • Joiner
  • Splitter
  • Utils
  • Constants
  • Text Utils
  • Video
  • Caption
  • Caption Enhancement
  • Caption Generation
  • Caption Preparation
  • Clipping
  • Clip Extraction Stages
  • Clip Frame Extraction
  • Transnetv2 Extraction
  • Video Frame Extraction
  • Embedding
  • Cosmos Embed1
  • Filtering
  • Clip Aesthetic Filter
  • Motion Filter
  • Motion Vector Backend
  • Io
  • Clip Writer
  • Video Reader
  • Preview
  • Preview
  • Tasks
  • Audio Task
  • Document
  • File Group
  • Image
  • Interleaved
  • Lance
  • Ocr
  • Sentinels
  • Tasks
  • Utils
  • Video
  • Utils
  • Atomic Io
  • Client Utils
  • Column Utils
  • Decoder Utils
  • File Utils
  • Gpu Utils
  • Grouping
  • Hash Utils
  • Hf Download Utils
  • Lance
  • Merge File Prefixes
  • Nvcodec Utils
  • Operation Utils
  • Performance Utils
  • Prompts
  • Ray Utils
  • Resumability Actor
  • Resumability Client
  • Retry Manifest
  • Split Large Files
  • Storage Utils
  • Vllm Utils
  • Windowing Utils
  • Writer Utils
  • Pipeline
  • ProcessingStage
  • CompositeStage
  • DocumentBatch
  • ImageBatch
  • VideoTask
  • AudioTask
  • XennaExecutor
  • Resources
On this page
  • Subpackages
  • Submodules
API ReferenceFull Library ReferenceNemo CuratorNemo Curator

nemo_curator.backends

||View as Markdown|

Subpackages

  • nemo_curator.backends.internal
  • nemo_curator.backends.ray_actor_pool
  • nemo_curator.backends.ray_data
  • nemo_curator.backends.xenna

Submodules

  • nemo_curator.backends.base
  • nemo_curator.backends.failed_task_markers
  • nemo_curator.backends.slurm_array
  • nemo_curator.backends.utils
Previous

nemo_curator

Next

nemo_curator.backends.base

NVIDIANVIDIA
Developer-friendly docs for your API
Privacy Policy | Your Privacy Choices | Terms of Service | Accessibility | Corporate Policies | Product Security | Contact

Copyright © 2026, NVIDIA Corporation.