Translation

Use NeMo Curator’s translation package to translate flat text fields or structured records, such as chat conversations stored under messages.*.content.

The experimental translation package is centered on TranslationStage, which composes segmentation, translation, reassembly, output formatting, and optional evaluation into a reusable text-processing stage.

Capabilities

• Translate a single text field such as text, a nested field path such as metadata.body, or wildcard paths such as messages.*.content
• Preserve machine-readable payloads, including valid JSON objects and arrays, instead of sending them to the translation model
• Emit translated output in replaced, raw, or both modes
• Emit segmented translation mappings for inspection or downstream evaluation
• Run FAITH scoring on translated text with FaithEvalFilter
• Score forward and reverse translation quality with TextQualityMetricStage

Before You Start

• Install translation extras as needed:
  • translation_common for basic translation support
  • translation_metrics for TextQualityMetricStage
  • translation_segmentation for segmentation_mode="fine"
  • translation_aws, translation_google, or translation_nmt for those backends
  • translation_all for the full translation feature set
• For backend_type="llm", configure an OpenAI-compatible async client. Refer to LLM Client Setup.
• If enable_faith_eval=True, configure an LLM client and scoring model even when translation itself uses a non-LLM backend.
• For non-LLM backends, use one of the built-in backend types: google, aws, or nmt.
• Input data is typically newline-delimited JSON with a text field or another field referenced through text_field.

Basic Translation Pipeline

The example below reads JSONL files, translates messages.*.content from English to Hindi, runs FAITH scoring, and writes the results back to JSONL.

1
import os
2
3
from nemo_curator.models.client.llm_client import GenerationConfig
4
from nemo_curator.models.client.openai_client import AsyncOpenAIClient
5
from nemo_curator.pipeline import Pipeline
6
from nemo_curator.stages.text.experimental.translation import TranslationStage
7
from nemo_curator.stages.text.io.reader import JsonlReader
8
from nemo_curator.stages.text.io.writer import JsonlWriter
9
10
client = AsyncOpenAIClient(
11
    api_key=os.environ["NVIDIA_API_KEY"],
12
    base_url="https://integrate.api.nvidia.com/v1",
13
    max_concurrent_requests=8,
14
)
15
16
pipeline = Pipeline(name="translate_chat_dataset")
17
pipeline.add_stage(JsonlReader(file_paths="input/*.jsonl"))
18
pipeline.add_stage(
19
    TranslationStage(
20
        client=client,
21
        model_name="openai/gpt-oss-120b",
22
        generation_config=GenerationConfig(max_tokens=2048),
23
        source_lang="en",
24
        target_lang="hi",
25
        text_field="messages.*.content",
26
        output_field="translated_text",
27
        output_mode="both",
28
        reconstruct_messages=True,
29
        enable_faith_eval=True,
30
        faith_threshold=2.5,
31
    )
32
)
33
pipeline.add_stage(JsonlWriter(path="translated/"))
34
results = pipeline.run()

source_lang and target_lang are required. Curator does not assume default translation languages.

Structured Translation Behavior

Structured translation works directly on nested records.

• Setting text_field="messages.*.content" extracts every message content string from the record
• Valid JSON objects and arrays are treated as non-translatable content and are preserved verbatim
• When output_mode="replaced", translated values are written back into the original field path
• When output_mode="raw" or output_mode="both", Curator emits translation_metadata with whole-text and segmented mappings

This makes the pipeline suitable for chat-style records where natural-language turns should be translated but tool payloads should remain untouched.

Segmentation and Output Control

TranslationStage exposes a few important controls:

• segmentation_mode="coarse" keeps line-level splitting with code-block awareness
• segmentation_mode="fine" uses sentence-level segmentation with structure preservation
• min_segment_chars bypasses segmentation for short text
• enable_faith_eval=True runs FAITH on exploded segment rows before reassembly, which avoids long-context scoring requests
• reconstruct_messages=True rebuilds translated message lists for structured chat-style inputs

TranslationStage Reference

All parameters are keyword-only. source_lang and target_lang are the only required values for non-LLM translation; LLM translation also requires client and model_name.

Input and Segmentation

Translation Inference

FAITH Evaluation

Output and Resume Controls

Validation and Option Interactions

• Empty or whitespace-only language codes are rejected.
• backend_type="llm" requires both client and a non-empty model_name, even in dry-run mode.
• FAITH always uses an LLM client. With Google, AWS, NMT, or a custom translation backend, pass a separate client and set faith_model_name.
• merge_scores=True with output_mode="replaced" is rejected. With FAITH disabled, score merging is skipped with a warning.
• output_mode="raw" removes output_field after building translation_metadata. Do not combine raw mode with message reconstruction: output_field is dropped before _build_translated_messages runs, so every translated_messages entry is an empty string. Use output_mode="both" when you need metadata and translated_messages.
• Custom prompt files and any local assets they reference must be available at the same absolute path on every worker.

DocumentBatch Walkthrough

TranslationStage operates on a DocumentBatch, which is a task wrapper around a pandas DataFrame or Arrow table.

The wrapper fields stay mostly constant across stages:

• task_id
• dataset_name
• _stage_perf
• _metadata

The main thing that changes is the DataFrame inside DocumentBatch.data.

Worked Example

Assume a single input row and this pipeline configuration:

1
TranslationStage(
2
    client=client,
3
    model_name="openai/gpt-oss-120b",
4
    text_field="text",
5
    source_lang="en",
6
    target_lang="hi",
7
    segmentation_mode="coarse",
8
    enable_faith_eval=True,
9
    output_mode="raw",
10
    merge_scores=True,
11
)

Input row:

id | text
7  | Explain grouped-query attention.
   | {"tool":"search","query":"GQA"}
   | ```python
   | print("hello")
   | ```
   | It reduces KV-cache memory.

1. SegmentationStage

The batch changes from one row per document to one row per translatable segment.

New columns:

• _seg_segments
• _seg_metadata
• _seg_doc_id

Example output:

id | _seg_segments
7  | Explain grouped-query attention.
7  | It reduces KV-cache memory.

Important details:

• Valid JSON payloads and fenced code blocks are not emitted as translatable segment rows.
• _seg_doc_id ties all segment rows back to the same source document.
• _seg_metadata is a JSON reconstruction template duplicated across the segment rows for that document.

For coarse segmentation, the metadata contains the original non-translated lines plus placeholders for translatable lines. For fine segmentation, it stores sentence-like units and their separators.

2. SegmentTranslationStage

The batch still has one row per segment, but each segment now gets its translated text and per-segment runtime/error data.

New columns:

• _translated
• _translation_time
• _translation_error

Example output:

id | _seg_segments                    | _translated
7  | Explain grouped-query attention. | समूहित-क्वेरी अटेंशन समझाइए।
7  | It reduces KV-cache memory.      | यह KV-cache मेमोरी को कम करता है।

3. FaithEvalFilter

When enable_faith_eval=True, FAITH runs on the exploded segment rows before reassembly.

New columns:

• faith_fluency
• faith_accuracy
• faith_idiomaticity
• faith_terminology
• faith_handling_of_format
• faith_avg
• faith_parse_failed

Each row is scored independently. Filtering does not happen yet, because dropping segment rows before reassembly would corrupt the reconstructed document.

4. ReassemblyStage

The batch collapses back to one row per original document by grouping on _seg_doc_id.

Removed internal columns:

• _seg_segments
• _seg_metadata
• _seg_doc_id
• _translated
• _translation_time
• _translation_error

Added document-level columns:

• translated_text
• translation_time
• translation_errors
• _translation_map
• _segmented_translation_map
• faith_segment_scores when FAITH is enabled
• aggregated faith_* columns when FAITH is enabled

Example output:

id | translated_text
7  | समूहित-क्वेरी अटेंशन समझाइए।
   | {"tool":"search","query":"GQA"}
   | ```python
   | print("hello")
   | ```
   | यह KV-cache मेमोरी को कम करता है।

Important details:

• translation_time is the sum of the segment-level translation times.
• translation_errors joins any non-empty segment errors.
• _translation_map and _segmented_translation_map are helper columns used later to build translation_metadata.
• When FAITH is enabled, reassembly averages the per-segment FAITH scores into document-level faith_* columns and writes the raw per-segment list to faith_segment_scores.
• For structured fields such as messages.*.content, reassembly writes translations back into the nested structure instead of only returning a flat string.

5. FaithThresholdFilterStage

When FAITH filtering is enabled, the threshold filter runs after reassembly on the aggregated FAITH score.

Rows with faith_avg < faith_threshold are dropped here. Rows with parse failures or no scored segments are preserved.

6. FormatTranslationOutputStage

When output_mode="raw" or output_mode="both", this stage builds:

• translation_metadata

translation_metadata contains:

• target_lang
• translation
• segmented_translation

When output_mode="raw", the final translated text column is dropped after metadata is constructed. This is useful when downstream consumers want metadata-rich output without keeping a separate top-level translated text column.

This stage also drops internal helper columns such as:

• _translation_map
• _segmented_translation_map

7. MergeFaithScoresStage

When merge_scores=True, FAITH scores are merged into the existing translation_metadata JSON under faith_scores.

At that point, the final row contains:

• the original source columns
• translation runtime and error columns
• FAITH score columns
• translation_metadata

Skip/Restore Path

When skip_translated=True, the pipeline inserts two additional stages:

• SkipExistingTranslationsStage
• RestoreSkippedRowsStage

SkipExistingTranslationsStage removes rows that already have a non-empty translation column from the DataFrame and stores them temporarily in DocumentBatch._metadata["_skipped_rows_state"].

RestoreSkippedRowsStage restores those rows later, fills in any missing score/metadata columns with defaults, and sorts them back into the original row order.

Skipped rows do not pass through segmentation, translation, FAITH scoring, or threshold filtering. They are restored after filtering, so pre-existing translations are never removed for having no new FAITH score. Missing columns created for newly translated rows are filled with safe defaults on restored rows, such as zero-valued FAITH scores and translation_metadata="{}".

Keep translation_column equal to output_field unless your input deliberately uses a separate completion marker. This ensures output formatting and skip detection refer to the same translation value.

Output modes apply after skipped rows are restored:

Quality Evaluation

FAITH Scoring

Enable FAITH scoring inside the translation pipeline when you want model-based adequacy checks on translated output:

1
TranslationStage(
2
    client=client,
3
    model_name="openai/gpt-oss-120b",
4
    source_lang="en",
5
    target_lang="de",
6
    text_field="text",
7
    enable_faith_eval=True,
8
    faith_threshold=2.5,
9
)

FAITH scores are merged into the output when output_mode="raw" or output_mode="both".

Non-LLM Translation with a Separate FAITH Model

The translation provider and FAITH scorer can be different services. This example translates with Google Cloud Translation v3 and scores the result with an OpenAI-compatible LLM:

1
import os
2
3
from nemo_curator.models.client.llm_client import GenerationConfig
4
from nemo_curator.models.client.openai_client import AsyncOpenAIClient
5
from nemo_curator.stages.text.experimental.translation import TranslationStage
6
7
faith_client = AsyncOpenAIClient(
8
    api_key=os.environ["NVIDIA_API_KEY"],
9
    base_url="https://integrate.api.nvidia.com/v1",
10
    max_concurrent_requests=64,
11
)
12
13
stage = TranslationStage(
14
    source_lang="en",
15
    target_lang="de",
16
    backend_type="google",
17
    backend_config={
18
        "api_version": "v3",
19
        "project_id": os.environ["GOOGLE_CLOUD_PROJECT"],
20
        "location": "global",
21
        "max_concurrent_requests": 32,
22
    },
23
    client=faith_client,  # Used for FAITH only; Google performs translation.
24
    enable_faith_eval=True,
25
    faith_model_name="openai/gpt-oss-120b",
26
    faith_generation_config=GenerationConfig(temperature=0.0, max_tokens=256),
27
    faith_max_concurrent_requests=64,
28
    faith_threshold=2.5,
29
    output_mode="both",
30
    merge_scores=True,
31
)

The client-side max_concurrent_requests limit and the stage-level faith_max_concurrent_requests semaphore both apply. The lower value is the effective concurrency, so size them together for the intended service limit.

Round-Trip Metrics

Backtranslation uses a second translation pass with reversed languages, followed by TextQualityMetricStage:

1
from nemo_curator.stages.text.experimental.translation import TextQualityMetricStage, TranslationStage
2
3
pipeline.add_stage(
4
    TranslationStage(
5
        client=client,
6
        model_name="openai/gpt-oss-120b",
7
        source_lang="hi",
8
        target_lang="en",
9
        text_field="translated_text",
10
        output_field="backtranslated_text",
11
        output_mode="both",
12
    )
13
)
14
pipeline.add_stage(
15
    TextQualityMetricStage(
16
        reference_text_field="text",
17
        hypothesis_text_field="backtranslated_text",
18
        metrics=[
19
            {"type": "sacrebleu", "threshold": 20.0},
20
            {"type": "chrf", "threshold": 40.0},
21
        ],
22
    )
23
)

Supported metric types are:

• sacrebleu
• chrf
• ter

Backend Selection

Use backend_type to switch between translation backends:

• llm: OpenAI-compatible async client
• google: Google translation backend
• aws: AWS translation backend
• nmt: NMT service backend

For non-LLM backends, pass backend-specific settings through backend_config.

Built-in Backend Configuration

Install the matching translation_google, translation_aws, or translation_nmt extra. Backend constructor errors surface during worker setup, and unknown backend names raise ValueError unless registered with register_backend().

Custom Prompts and Dry Runs

Translation and FAITH prompts are YAML mappings with required top-level system and user keys. Custom files must use absolute paths.

The translation user template can reference:

• {source_lang}: full source-language name
• {target_lang}: full target-language name
• {src}: source segment

The FAITH templates can reference {source_language}, {target_language}, {source_text}, and {translated_text}. Escape literal braces for Python string formatting by doubling them, such as {{"score": 5}}.

Example translation prompt:

1
system: You are a domain-specific technical translator.
2
user: |-
3
  Translate this {source_lang} text to {target_lang}.
4
  Preserve API names and code exactly.
5
  Return only the translation wrapped in 〘 and 〙.
6
7
  {src}

Validate prompt expansion without sending translation requests:

1
stage = TranslationStage(
2
    source_lang="en",
3
    target_lang="ja",
4
    client=client,
5
    model_name="openai/gpt-oss-120b",
6
    translation_prompt_path="/shared/prompts/technical-translation.yaml",
7
    dry_run=True,
8
    dry_run_log_count=3,
9
    health_check=False,
10
)

Dry-run mode still validates required constructor fields, loads the prompt, and initializes the client or backend during worker setup. Set health_check=False when the dry run must not issue an LLM health-check request. The stage logs up to dry_run_log_count expanded LLM messages (or source-segment previews for non-LLM backends), writes empty translations, and records zero translation time and no translation errors.

Notes

• The translation package is designed for pipeline execution. Avoid converting large datasets to pandas on the driver just to orchestrate translation.
• For structured inputs, wildcard paths and nested paths are first-class inputs to the library. You do not need to flatten records manually before calling TranslationStage.