Installation Guide

This guide covers installing NeMo Curator with support for all modalities and verifying your installation is working correctly.

Before You Start

System Requirements

For comprehensive system requirements and production deployment specifications, refer to Production Deployment Requirements.

Quick Start Requirements:

• OS: Ubuntu 24.04/22.04/20.04 (recommended)
• Python: 3.10, 3.11, or 3.12
• Memory: 16GB+ RAM for basic text processing
• GPU (optional): NVIDIA GPU with 16GB+ VRAM for acceleration
• CUDA 12 (required for audio_cuda12, video_cuda12, image_cuda12, and text_cuda12 extras)

Development vs Production

Installation Methods

Choose one of the following installation methods based on your needs:

Install FFmpeg and Encoders (Required for Video)

Curator’s video pipelines rely on FFmpeg for decoding and encoding. If you plan to encode clips (for example, using --transcode-encoder libopenh264 or h264_nvenc), install FFmpeg with the corresponding encoders.

$
curl -fsSL https://raw.githubusercontent.com/NVIDIA-NeMo/Curator/main/docker/common/install_ffmpeg.sh -o install_ffmpeg.sh
$
chmod +x install_ffmpeg.sh
$
sudo bash install_ffmpeg.sh

Package Extras

NeMo Curator provides several installation extras to install only the components you need:

Installation Verification

After installation, verify that NeMo Curator is working correctly:

1. Basic Import Test

1
# Test basic imports
2
import nemo_curator
3
print(f"NeMo Curator version: {nemo_curator.__version__}")
4
5
# Test core modules
6
from nemo_curator.pipeline import Pipeline
7
from nemo_curator.tasks import DocumentBatch
8
print("✓ Core modules imported successfully")

2. GPU Availability Check

If you installed GPU support, verify GPU access:

1
# Check GPU availability
2
try:
3
    import torch
4
    if torch.cuda.is_available():
5
        print(f"✓ GPU available: {torch.cuda.get_device_name(0)}")
6
        print(f"✓ GPU memory: {torch.cuda.get_device_properties(0).total_memory / 1e9:.1f} GB")
7
    else:
8
        print("⚠ No GPU detected")
9
    
10
    # Check cuDF for GPU deduplication
11
    import cudf
12
    print("✓ cuDF available for GPU-accelerated deduplication")
13
except ImportError as e:
14
    print(f"⚠ Some GPU modules not available: {e}")

3. Run a Quickstart Tutorial

Try a modality-specific quickstart to see NeMo Curator in action:

• Text Curation Quickstart - Set up and run your first text curation pipeline
• Audio Curation Quickstart - Get started with audio dataset curation
• Image Curation Quickstart - Curate image-text datasets for generative models
• Video Curation Quickstart - Split, encode, and curate video clips at scale