Nemotron-Parse PDF Pipeline

Convert PDF datasets into interleaved Parquet output using NVIDIA’s Nemotron-Parse vision-language model. Unlike traditional text-only PDF parsers, Nemotron-Parse extracts text, images, and reading order in one pass — producing rows directly compatible with the interleaved dataset format.

How it Works

NemotronParsePDFReader is a composite stage that expands into four underlying sub-stages:

1. PDFPartitioningStage — reads a JSONL manifest of PDF entries and packs them into FileGroupTask objects.
2. PDFPreprocessStage — extracts PDF bytes from the configured source, renders pages to images with scale-to-fit safeguarding against OOM on large pages.
3. NemotronParseInferenceStage — runs Nemotron-Parse via vLLM (recommended) or Hugging Face Transformers, with text_in_pic and enforce_eager flags and free-port retry on collisions.
4. NemotronParsePostprocessStage — parses model output, aligns images and captions, crops images, and emits the final interleaved rows.

The output is interleaved Parquet ready to be filtered with Interleaved Filters and written to MINT-1T-style WebDataset shards.

Before You Start

Choose your PDF source and confirm the prerequisites:

• GPU: Required. Nemotron-Parse runs on GPU via vLLM (recommended) or Hugging Face Transformers.
• vLLM: Strongly recommended for throughput. Falls back to HF Transformers if backend="hf" is set.
• pypdfium2: Required Python dependency for PDF rendering. Installed automatically with the interleaved_cpu or interleaved_cuda12 extras (e.g., uv sync --extra interleaved_cuda12).
• Manifest: A JSONL file listing the PDFs to process. Each line should specify the PDF location relative to the source directory you choose.

Choosing a PDF Source

Pass exactly one of pdf_dir, zip_base_dir, or jsonl_base_dir so the preprocess stage knows where to find the PDF bytes:

Backend Selection

The inference stage retries transient port collisions when starting vLLM. Non-retryable startup failures, such as invalid configuration or GPU out-of-memory errors, fail immediately.

Usage

A minimal end-to-end pipeline that reads PDFs from a directory and writes interleaved Parquet:

1
from nemo_curator.pipeline import Pipeline
2
from nemo_curator.backends.xenna import XennaExecutor
3
from nemo_curator.stages.interleaved.pdf.nemotron_parse import NemotronParsePDFReader
4
from nemo_curator.stages.interleaved.io.writers.tabular import InterleavedParquetWriter
5
6
pipeline = Pipeline(name="pdf_to_interleaved")
7
8
# 1. Parse PDFs into interleaved rows
9
pipeline.add_stage(
10
    NemotronParsePDFReader(
11
        manifest_path="./pdfs.jsonl",
12
        pdf_dir="/data/pdfs",
13
        backend="vllm",
14
        pdfs_per_task=10,
15
        max_pages=50,
16
        inference_batch_size=4,
17
    )
18
)
19
20
# 2. Write interleaved Parquet
21
pipeline.add_stage(InterleavedParquetWriter(output_dir="./parsed_pdfs"))
22
23
executor = XennaExecutor()
24
pipeline.run(executor)

For executor options and configuration, refer to Execution Backends.

Example: CC-MAIN PDF Dump

Parse a Common Crawl PDF dump from its zip hierarchy:

1
NemotronParsePDFReader(
2
    manifest_path="./cc_pdfs.jsonl",
3
    zip_base_dir="/data/CC-MAIN-2021-31-PDF-UNTRUNCATED",
4
    backend="vllm",
5
    file_names_field="cc_pdf_file_names",
6
    pdfs_per_task=20,
7
)

Example: JSONL-Encoded PDFs

Parse a JSONL-encoded dataset (e.g., GitHub-hosted PDFs where each line contains the bytes):

1
NemotronParsePDFReader(
2
    manifest_path="./github_pdfs.jsonl",
3
    jsonl_base_dir="/data/github_pdfs",
4
    backend="vllm",
5
)

Parameters

Tune the vLLM Engine

NemotronParseInferenceStage.engine_kwargs is None by default. It passes additional settings to NeMo Curator’s shared vLLM initializer: vLLM engine settings are forwarded to vllm.LLM, while helper settings such as max_port_retries control initialization itself. Use this field when you compose the pipeline from individual stages and need controls that are not exposed by NemotronParsePDFReader:

1
from nemo_curator.pipeline import Pipeline
2
from nemo_curator.stages.interleaved.io import InterleavedParquetWriterStage
3
from nemo_curator.stages.interleaved.pdf.nemotron_parse import (
4
    NemotronParseInferenceStage,
5
    NemotronParsePostprocessStage,
6
    PDFPartitioningStage,
7
    PDFPreprocessStage,
8
)
9
10
pipeline = Pipeline(name="tuned_nemotron_parse")
11
pipeline.add_stage(PDFPartitioningStage(manifest_path="./pdfs.jsonl", pdfs_per_task=10))
12
pipeline.add_stage(PDFPreprocessStage(pdf_dir="/data/pdfs", max_pages=50))
13
pipeline.add_stage(
14
    NemotronParseInferenceStage(
15
        backend="vllm",
16
        max_num_seqs=64,
17
        engine_kwargs={
18
            "gpu_memory_utilization": 0.90,
19
            "max_num_batched_tokens": 16384,
20
        },
21
    )
22
)
23
pipeline.add_stage(NemotronParsePostprocessStage(min_crop_px=10))
24
pipeline.add_stage(
25
    InterleavedParquetWriterStage(
26
        path="./parsed_pdfs",
27
        materialize_on_write=False,
28
    )
29
)

The installed vLLM version performs final validation of forwarded engine settings. Check the matching vLLM documentation before using additional keys.

Common tuning controls include:

Ray Data Fanout

PDFPartitioningStage is a Ray Data fanout stage. It reads the manifest on one worker, emits one FileGroupTask per pdfs_per_task group, and Ray Data repartitions the result to one emitted task per block. Downstream preprocess and inference stages can then consume those blocks in parallel instead of receiving the whole manifest as one block.

This behavior is automatic with RayDataExecutor; no stage-spec override is required. pdfs_per_task still controls the work in each emitted task:

• Lower values create more tasks and expose more downstream parallelism, with more scheduling overhead.
• Higher values reduce scheduling overhead but can leave GPUs idle when the number of tasks is smaller than the available workers.
• max_pdfs is applied before tasks are created, so it remains useful for small validation runs.

Xenna uses its own task dispatch and does not consume the Ray Data fanout marker.

Output Format

Each output row represents a single item (text, image, or metadata) from a parsed PDF page. Rows sharing a sample_id belong to the same document. Example output JSON:

1
{
2
  "sample_id": "doc_42",
3
  "position": 0,
4
  "modality": "text",
5
  "text_content": "# Introduction\n\nThis paper investigates...",
6
  "binary_content": null,
7
  "source_files": ["pdf_42.pdf"],
8
  "url": "https://example.com/pdf_42.pdf"
9
}
10
{
11
  "sample_id": "doc_42",
12
  "position": 1,
13
  "modality": "image",
14
  "text_content": null,
15
  "binary_content": "<bytes>",
16
  "source_files": ["pdf_42.pdf"]
17
}
18
{
19
  "sample_id": "doc_42",
20
  "position": 2,
21
  "modality": "text",
22
  "text_content": "Figure 1 shows the architecture...",
23
  "binary_content": null,
24
  "source_files": ["pdf_42.pdf"]
25
}

Output Schema

The output is directly compatible with Interleaved IO readers and writers — the schema matches INTERLEAVED_SCHEMA exactly.

Inspect Inference Metrics

NemotronParseInferenceStage records additive custom metrics on each output task. Aggregate the final pipeline results with TaskPerfUtils:

1
import time
2
3
from nemo_curator.tasks.utils import TaskPerfUtils
4
5
started = time.perf_counter()
6
results = pipeline.run(executor)
7
wall_time_s = time.perf_counter() - started
8
9
metrics = TaskPerfUtils.aggregate_task_metrics(results, prefix="task")
10
metric_prefix = "task_nemotron_parse_inference_custom"
11
12
valid_pages = metrics.get(f"{metric_prefix}.num_valid_pages_sum", 0.0)
13
output_tokens = metrics.get(f"{metric_prefix}.total_output_tokens_sum", 0.0)
14
15
pages_per_second = valid_pages / wall_time_s if wall_time_s else 0.0
16
output_tokens_per_second = output_tokens / wall_time_s if wall_time_s else 0.0
17
18
print(f"{pages_per_second:.2f} pages/s")
19
print(f"{output_tokens_per_second:.2f} output tokens/s")

aggregate_task_metrics() appends _sum, _mean, and _std to each flattened metric. Use _sum for additive counts and total durations. Measure end-to-end throughput against pipeline wall time rather than the sum of per-task inference times, because tasks can run concurrently.

Metric Reference

Use the quality signals alongside throughput. A high num_output_length_truncated value means outputs are reaching the stage’s built-in 9,000-token generation limit and warrants inspection of those pages; num_empty_outputs and num_skipped_pages identify model-output and image-decoding failures that raw pages-per-second figures can hide.

vLLM Retry Behavior

There are two separate retry paths:

1. Engine startup: create_vllm_llm() chooses a new MASTER_PORT and retries up to max_port_retries=3 times for direct address-in-use errors or vLLM v1’s wrapped Engine core initialization failed error. Retries wait two to five seconds with jitter. Known non-retryable failures, including out-of-memory, device-side assertion, and invalid configuration errors, are raised immediately.
2. Inference: vLLM generation is attempted up to three times. After a failed attempt, the stage resets the engine before retrying. Successful retries contribute to the vllm_retries task metric; the final exception is raised after the third failed attempt.

If startup retries are exhausted, first check the worker log for the original failure. For repeated port collisions, reduce the number of vLLM replicas starting simultaneously or set a larger stage-level max_port_retries through engine_kwargs. Do not mask CUDA out-of-memory or invalid-model errors by increasing retries; tune memory-related engine settings or correct the model configuration instead.

Render Timeout

The preprocess stage replaces signal.SIGALRM with a multiprocessing fork-based timeout (_RENDER_TIMEOUT_S = 60 by default). This is required because Xenna runs stage workers inside Ray actor processes on non-main threads, where SIGALRM raises ValueError: signal only works in main thread. The forked child inherits the PDF bytes via copy-on-write and is killed if it exceeds the timeout, reliably escaping any hung C-extension code inside pypdfium2.

You don’t need to configure this — it works automatically. If you find legitimate PDFs that take longer than 60 seconds to render, the constant lives at nemo_curator/stages/interleaved/pdf/nemotron_parse/preprocess.py.

Benchmarking

A standalone benchmark script ships at benchmarking/scripts/nemotron_parse_pdf_benchmark.py. It uses the same TaskPerfUtils aggregation shown above and reports end-to-end pages per second and output tokens per second. Use a representative manifest and the public tutorial arguments to compare configurations before scaling to your full corpus; the repository’s nightly benchmark orchestration is not required.

Best Practices

• Use vLLM unless you can’t: the vllm backend is substantially faster than hf. Only fall back to hf for debugging or in environments where vLLM is unavailable.
• Cap max_pages for outliers: very long PDFs (1000+ pages) can dominate runtime. The default 50 pages handles most academic papers and articles; raise to 200+ for book-length sources.
• Tune pdfs_per_task for parallelism: smaller values (5–10) parallelize better across many GPUs; larger values (20–50) reduce per-task overhead on smaller clusters.
• Set enforce_eager=True in restricted environments: vLLM’s torch.compile path can fail on certain hosts. Disabling compilation trades throughput for compatibility.
• Pair with interleaved filters: PDF parsing produces noisy output. Chain with the Interleaved Filters (blur, CLIP score) to drop low-quality samples before training.

Related Topics

• Interleaved IO — readers and writers that consume the Parquet output of this pipeline.
• Interleaved Filters — sample-level filters to apply after parsing.
• Common Crawl — companion source for web-scale PDF input via CC-MAIN dumps.