Nemotron OCR Synthetic Data

The Nemotron OCR pipeline converts images into word-level OCR annotations and, optionally, scored visual question-answering conversations. It runs NemotronOCR-v2 locally on a GPU, can ask Nemotron-Nano-Omni through the NVIDIA Inference API to verify every bounding box, and writes one JSONL record per image.

Runtime architecture

The included runner starts a Ray cluster and uses XennaExecutor; it does not expose a Ray Data backend option. All workers must see the extracted image directory and output path through the same filesystem.

Installation and model access

Use the NeMo Curator container for this pipeline. In addition to CUDA and the normal SDG/image dependencies, the repository Dockerfile builds the nemotron-ocr C++/CUDA extension from the nvidia/nemotron-ocr-v2 repository. Installing sdg_cuda12 or image_cuda12 alone does not install that extension.

$
docker pull nvcr.io/nvidia/nemo-curator:{{ container_version }}
$
$
docker run --gpus all -it --rm \
>
  -v "$PWD:/workspace" \
>
  -w /workspace \
>
  nvcr.io/nvidia/nemo-curator:{{ container_version }}

Inside the container:

$
source /opt/venv/env.sh
$
python -c "from nemotron_ocr.inference.pipeline_v2 import NemotronOCRV2; print('Nemotron OCR ready')"

The default OCR stage downloads nvidia/nemotron-ocr-v2 from Hugging Face and uses its v2_multilingual subdirectory. Mount a persistent Hugging Face cache, or pass a pre-downloaded v2_multilingual directory with --nemotron-model-dir.

The repository Dockerfile compiles for CUDA architectures 8.0, 8.6, 8.9, and 9.0. GPU memory requirements depend on image size, model version, and worker concurrency; the stage does not enforce a VRAM minimum.

The optional verifier reads NVINFERENCE_API_KEY from each worker’s environment during setup. Do not place the key in source, CLI arguments, or serialized stage configuration. Obtain a key from NVIDIA Build and ensure Ray workers inherit it.

Input sources

HFDatasetImageReaderStage supports three source forms:

The image column may contain a PIL image, raw bytes, a Hugging Face Image dictionary with bytes or path, or a valid file-path string. Images are converted to RGB and saved as <image_dir>/<image_id>.jpg.

If id_column is unset, zero-padded row indexes are used. If set, only the first row for each ID is processed. IDs must be safe as local filenames. Hub and image-folder limits apply before ID deduplication, so the final number of unique tasks can be lower than the limit.

OCR data model

OCRData extends ImageTaskData and carries the reader, OCR, and verifier fields. When OCRScoringQAStage runs, it converts the payload to OCRConversationData, an OCRData subclass that adds the conversation field while retaining the base fields:

Each OCRDenseItem contains bbox_2d, text_content, optional quad, valid, bbox_match, and text_errors. Nemotron output is scaled to integer 0–1000 coordinates, and inverted vertical bounds are sorted.

Bounding-box verification

OCRScoringQAStage makes one remote verifier request per image:

The verifier returns ocr_mode, indexed box scores, and missing_text. Missing or nonnumeric scores invalidate a box. If every original box is invalid, the image is invalid. Empty or unparseable responses also invalidate the image.

NVInferenceClient uses the NVIDIA integration endpoint, reads NVINFERENCE_API_KEY, streams final-answer content, allows 10 concurrent requests per worker, uses a 120-second timeout, and retries rate-limit and connection failures three times with exponential backoff and jitter.

QA interaction shapes

The source defines five QA type constants. Four tag and balance the multi-turn families; the fifth, dense_dump, names the separate single-turn all-words path. Single-occurrence and repeated-text routing within the four tagged families produces six user-visible shapes:

Generation is seeded from the framework task ID. At most 100 multi-turn QA pairs are balanced across the four tagged families. When five or more OCR items are invalid, text-to-location questions are disabled.

When no text is missing, dense_dump_prob can select a single turn listing every valid word. It combines one of 33 instruction phrasings with one of 11 answer formats:

Conversation schema

Within each output JSONL record, the top-level conversation field contains a wrapper object whose inner conversation field is the ordered turn list:

1
{
2
  "conversation": {
3
    "conversation": [
4
      {"sender": "user", "fragments": [{"t": "image", "value": "000042.jpg"}, "What text is in the bounding box [120, 90, 440, 160]?"]},
5
      {"sender": "assistant", "fragments": ["PROJECT STATUS"]},
6
      {"sender": "user", "fragments": ["Point at the text 'TOTAL'. Answer with [x, y]."]},
7
      {"sender": "assistant", "fragments": ["[732, 518]"]}
8
    ]
9
  }
10
}

The image value is the cached filename, not its bytes. Use --image-parent to make image_path portable and distribute the cached images with the JSONL.

Failure and restart model

• Cached JPEGs are reused by filename without verifying their content.
• OCR catches exceptions per image; prompt/response errors are also isolated per image. A generation or response-count failure can invalidate a whole scoring batch.
• Remote failures become empty responses after retries and invalidate the affected image.
• Worker JSONL records are flushed immediately. valid_only=True skips invalid tasks.
• The CLI defaults to valid_only=False, retaining invalid records and their error field.
• The writer removes is_valid and all None fields but retains empty lists, strings, and false.
• There is no processed-ID checkpoint. Reruns repeat OCR and API calls even when JPEGs are cached.
• Stale *_worker*.jsonl files are not cleaned before a run. Remove them or use a fresh output basename after interruption.

Continue with the runnable tutorial.

Related topics

• Synthetic data generation
• LLM client configuration
• Image curation
• Execution backends