> For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/nemo/curator/v26.04/curate-images/process-data/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/curator/_mcp/server. # Process Data ## Docs - [Overview](https://docs.nvidia.com/nemo/curator/v26.04/curate-images/process-data/embeddings.md): Generate image embeddings using built-in embedders for classification, filtering, and similarity search - [CLIP Embedder](https://docs.nvidia.com/nemo/curator/v26.04/curate-images/process-data/embeddings/clip-embedder.md): Generate CLIP embeddings for images using OpenAI's ViT-L/14 model for downstream classification and filtering tasks - [Overview](https://docs.nvidia.com/nemo/curator/v26.04/curate-images/process-data/filters.md): Image filtering tools including aesthetic and NSFW filters for dataset quality control - [Aesthetic Filter](https://docs.nvidia.com/nemo/curator/v26.04/curate-images/process-data/filters/aesthetic.md): Aesthetic filter for predicting visual quality of images using CLIP embeddings and human preference training - [NSFW Filter](https://docs.nvidia.com/nemo/curator/v26.04/curate-images/process-data/filters/nsfw.md): NSFW filter for detecting inappropriate content in images using CLIP embeddings and MLP architecture > **Note:** This page contains both a page directory (above) and the landing page content (below). The page directory is generated for agent use and does not appear on the landing page. > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/nemo/curator/v26.04/curate-images/process-data/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/curator/_mcp/server. > Process image data using embeddings, filters, and filtering for high-quality dataset curation # Process Data for Image Curation Process image data you've loaded from tar archives using NeMo Curator's suite of tools. These tools help you generate embeddings, filter images, and prepare your dataset to produce high-quality data for downstream AI tasks such as generative model training, dataset analysis, or quality control. ## How it Works Image processing in NeMo Curator follows a pipeline-based approach with these stages: 1. **Partition files** using `FilePartitioningStage` to distribute tar files 2. **Read images** using `ImageReaderStage` with DALI acceleration 3. **Generate embeddings** using `ImageEmbeddingStage` with CLIP models 4. **Apply filters** using `ImageAestheticFilterStage` and `ImageNSFWFilterStage` 5. **Save results** using `ImageWriterStage` to export curated datasets Each stage processes `ImageBatch` objects containing images, metadata, and processing results. You can use built-in stages or create custom stages for advanced use cases. --- ## Embedding Options #### [CLIP Embedding Stage](/curate-images/process-data/embeddings/clip-embedder) Generate image embeddings using CLIP models with GPU acceleration. Supports various CLIP architectures and automatic model downloading. ImageEmbeddingStage CLIP GPU-accelerated ## Filter Options #### [Aesthetic Filter Stage](/curate-images/process-data/filters/aesthetic) Assess the subjective quality of images using a model trained on human aesthetic preferences. Filters images based on aesthetic score thresholds. ImageAestheticFilterStage aesthetic\_score #### [NSFW Filter Stage](/curate-images/process-data/filters/nsfw) Detect not-safe-for-work (NSFW) content in images using a CLIP-based filter. Filters explicit material from your datasets. ImageNSFWFilterStage nsfw\_score