> This page is for version Latest · v1.4.0 (26.09) (default).
> For other versions, use one of these documentation indexes:
> - Latest · v1.4.0 (26.09) (default): https://docs.nvidia.com/nemo/curator/latest/llms.txt
> - Main · preview: https://docs.nvidia.com/nemo/curator/main/llms.txt
> - 26.09 · v1.4.0: https://docs.nvidia.com/nemo/curator/v26.09/llms.txt
> - 26.07 · v1.3.0: https://docs.nvidia.com/nemo/curator/v26.07/llms.txt
> - 26.04 · v1.2.0: https://docs.nvidia.com/nemo/curator/v26.04/llms.txt
> - 26.02 · v1.1.0: https://docs.nvidia.com/nemo/curator/v26.02/llms.txt
> - 25.09 · v1.0.0: https://docs.nvidia.com/nemo/curator/v25.09/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/curator/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/curator/_mcp/server.

# nemo_curator.eval.llm_judge.workflow

Config-driven text LLM judge workflow, run through a NeMo Curator pipeline.

The input records may have any text schema. The Jinja templates and score
rubrics in the judge config define which fields are evaluated, what the
judge returns, and how judges are grouped into `execution.stages`, each of
which runs as its own NDD stage.

## Module Contents

### Classes

| Name                                                                         | Description                                  |
| ---------------------------------------------------------------------------- | -------------------------------------------- |
| [`LLMJudgeWorkflow`](#nemo_curator-eval-llm_judge-workflow-LLMJudgeWorkflow) | End-to-end config-driven LLM judge workflow. |

### Functions

| Name                                                                                                 | Description                                                                                 |
| ---------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------- |
| [`_build_filter_stages`](#nemo_curator-eval-llm_judge-workflow-_build_filter_stages)                 | Build Curator filters that retain rows satisfying every configured condition.               |
| [`_build_language_filter_stage`](#nemo_curator-eval-llm_judge-workflow-_build_language_filter_stage) | Build an optional FastText language gate without retaining its score column.                |
| [`_keep_judge_score`](#nemo_curator-eval-llm_judge-workflow-_keep_judge_score)                       | Return whether one NDD judge result satisfies a declarative comparison.                     |
| [`_load_yaml`](#nemo_curator-eval-llm_judge-workflow-_load_yaml)                                     | -                                                                                           |
| [`_place_filters`](#nemo_curator-eval-llm_judge-workflow-_place_filters)                             | Place top-level filters after the NDD stage that produces their judge column.               |
| [`_read_template`](#nemo_curator-eval-llm_judge-workflow-_read_template)                             | -                                                                                           |
| [`_start_inference_server`](#nemo_curator-eval-llm_judge-workflow-_start_inference_server)           | Start all configured Dynamo models behind one OpenAI-compatible endpoint.                   |
| [`_validate_filter_references`](#nemo_curator-eval-llm_judge-workflow-_validate_filter_references)   | Ensure filters refer to a configured judge output column and rubric score.                  |
| [`build_config_builder`](#nemo_curator-eval-llm_judge-workflow-build_config_builder)                 | Build one NDD configuration for a selected group of judge columns.                          |
| [`build_pipeline`](#nemo_curator-eval-llm_judge-workflow-build_pipeline)                             | Build a streaming pipeline with an optional language gate, NDD stages, filters, and writer. |

### Data

[`DataFormat`](#nemo_curator-eval-llm_judge-workflow-DataFormat)

[`FilterOperator`](#nemo_curator-eval-llm_judge-workflow-FilterOperator)

[`_FILTER_OPERATORS`](#nemo_curator-eval-llm_judge-workflow-_FILTER_OPERATORS)

### API

```python
class nemo_curator.eval.llm_judge.LLMJudgeWorkflow(
    judge_config: str | pathlib.Path,
    input_path: str,
    output_path: str,
    input_format: nemo_curator.eval.llm_judge.workflow.DataFormat = 'jsonl',
    output_format: nemo_curator.eval.llm_judge.workflow.DataFormat = 'jsonl',
    files_per_partition: int | None = None,
    language: str | None = None,
    fasttext_langid_model_path: str | None = None,
    min_langid_score: float = 0.3,
    language_text_field: str = 'raw_text',
    checkpoint_path: str | None = None
)
```

Dataclass

**Bases:** [WorkflowBase](/nemo/curator/nemo-curator/nemo_curator/pipeline/workflow#nemo_curator-pipeline-workflow-WorkflowBase)

End-to-end config-driven LLM judge workflow.

Loads a judge config YAML (models, Jinja prompt templates, score rubrics,
and `execution.stages`), starts a Dynamo/vLLM inference server hosting
the configured judge models, then runs one Curator pipeline containing:
reader -> optional FastText language gate -> one NDD `DataDesignerStage`
(+ its filters) per judge stage -> writer.

**`checkpoint_path`** `str | None = None`

---

**`config`** `dict[str, object] = field(init=False)`

---

**`config_path`** `Path = field(init=False)`

---

**`fasttext_langid_model_path`** `str | None = None`

---

**`files_per_partition`** `int | None = None`

---

**`input_format`** `DataFormat = 'jsonl'`

---

**`input_path`** `str`

---

**`judge_config`** `str | Path`

---

**`language`** `str | None = None`

---

**`language_text_field`** `str = 'raw_text'`

---

**`min_langid_score`** `float = 0.3`

---

**`output_format`** `DataFormat = 'jsonl'`

---

**`output_path`** `str`

---

```python
nemo_curator.eval.llm_judge.LLMJudgeWorkflow.__post_init__() -> None
```

```python
nemo_curator.eval.llm_judge.LLMJudgeWorkflow._build_judge_stages(
    endpoint: str
) -> list[tuple[str, data_designer.config.DataDesignerConfigBuilder, list[data_designer.config.ModelProvider], dict[str, object] | None, int | None, list[dict[str, object]]]]
```

```python
nemo_curator.eval.llm_judge.LLMJudgeWorkflow.run() -> nemo_curator.pipeline.workflow.WorkflowRunResult
```

Run the complete LLM judge pipeline.

**Returns:** [`WorkflowRunResult`](/nemo/curator/nemo-curator/nemo_curator/pipeline/workflow#nemo_curator-pipeline-workflow-WorkflowRunResult)

WorkflowRunResult containing the pipeline output tasks and timing metadata.

```python
nemo_curator.eval.llm_judge.workflow._build_filter_stages(
    filters: list[dict[str, object]],
    name_prefix: str
) -> list[nemo_curator.stages.text.filters.Filter]
```

Build Curator filters that retain rows satisfying every configured condition.

```python
nemo_curator.eval.llm_judge.workflow._build_language_filter_stage(
    language: str | None,
    model_path: str | None,
    min_score: float,
    text_field: str
) -> nemo_curator.stages.text.filters.ScoreFilter | None
```

Build an optional FastText language gate without retaining its score column.

```python
nemo_curator.eval.llm_judge.workflow._keep_judge_score(
    judge_result: object,
    score_name: str,
    operator: nemo_curator.eval.llm_judge.workflow.FilterOperator,
    expected: object
) -> bool
```

Return whether one NDD judge result satisfies a declarative comparison.

```python
nemo_curator.eval.llm_judge.workflow._load_yaml(
    path: pathlib.Path
) -> dict[str, object]
```

```python
nemo_curator.eval.llm_judge.workflow._place_filters(
    config: dict[str, object],
    stages: list[dict[str, object]]
) -> list[list[dict[str, object]]]
```

Place top-level filters after the NDD stage that produces their judge column.

```python
nemo_curator.eval.llm_judge.workflow._read_template(
    path: str,
    config_path: pathlib.Path
) -> str
```

```python
nemo_curator.eval.llm_judge.workflow._start_inference_server(
    config: dict[str, object],
    models: list[dict[str, object]],
    config_path: pathlib.Path
) -> nemo_curator.core.serve.InferenceServer
```

Start all configured Dynamo models behind one OpenAI-compatible endpoint.

```python
nemo_curator.eval.llm_judge.workflow._validate_filter_references(
    config: dict[str, object],
    stages: list[dict[str, object]]
) -> None
```

Ensure filters refer to a configured judge output column and rubric score.

```python
nemo_curator.eval.llm_judge.workflow.build_config_builder(
    config_path: str | pathlib.Path,
    endpoint: str,
    models: list[dict[str, object]],
    judges: list[dict[str, object]]
) -> tuple[data_designer.config.DataDesignerConfigBuilder, list[data_designer.config.ModelProvider]]
```

Build one NDD configuration for a selected group of judge columns.

```python
nemo_curator.eval.llm_judge.workflow.build_pipeline(
    input_path: str,
    input_format: nemo_curator.eval.llm_judge.workflow.DataFormat,
    output_path: str,
    output_format: nemo_curator.eval.llm_judge.workflow.DataFormat,
    judge_stages: list[tuple[str, data_designer.config.DataDesignerConfigBuilder, list[data_designer.config.ModelProvider], dict[str, object] | None, int | None, list[dict[str, object]]]],
    language_filter_stage: nemo_curator.stages.text.filters.ScoreFilter | None,
    files_per_partition: int | None
) -> nemo_curator.pipeline.Pipeline
```

Build a streaming pipeline with an optional language gate, NDD stages, filters, and writer.

```python
nemo_curator.eval.llm_judge.workflow.DataFormat = Literal['jsonl', 'parquet']
```

```python
nemo_curator.eval.llm_judge.workflow.FilterOperator = Literal['eq', 'ne', 'gt', 'gte', 'lt', 'lte', 'in', 'not_in']
```

```python
nemo_curator.eval.llm_judge.workflow._FILTER_OPERATORS = {'eq', 'ne', 'gt', 'gte', 'lt', 'lte', 'in', 'not_in'}
```