Skip to main content

Ctrl+K

NeMo-Curator

GitHub

NeMo-Curator

GitHub

Table of Contents

Home

About NeMo Curator

Overview of NeMo Curator
Key Features
Concepts
NeMo Curator Release Notes: 25.09

Get Started

About Getting Started
Text Curation Quickstart
Image Curation Quickstart
Video Curation Quickstart
Audio Curation Quickstart

Curate Text

About Text Curation
Tutorials
Load Data
Process Data

Curate Images

About Image Curation
Tutorials
- Beginner Tutorial
- Image Duplicate Removal Workflow
Load Data
- Tar Archives
Process Data
- Filters
  - Aesthetic Filter
  - NSFW Filter
- Embeddings
  - CLIP ImageEmbeddingStage
Save and Export

Curate Video

About Video Curation
Tutorials
Load Data
Process Data
Save & Export

Curate Audio

About Audio Curation
Tutorials
- Beginner Tutorial
Load Data
Process Data
Save & Export

Setup & Deployment

About Setup & Deployment
Install Curator

Reference

About References
Infrastructure
API Reference
Tools

API Reference
stages
stages.deduplication
stages.deduplication.shuffle_utils

`stages.deduplication.shuffle_utils`#

Submodules#

stages.deduplication.shuffle_utils.rapidsmpf_shuffler
stages.deduplication.shuffle_utils.stage

previous

stages.deduplication.semantic.workflow

next

stages.deduplication.shuffle_utils.rapidsmpf_shuffler

On this page

Submodules

Privacy Policy | Manage My Privacy | Do Not Sell or Share My Data | Terms of Service | Accessibility | Corporate Policies | Product Security | Contact

Copyright © 2025 NVIDIA Corporation.