nemo_curator.eval.llm_judge

Submodules

nemo_curator.eval.llm_judge.workflow

Package Contents

Classes

Name	Description
`LLMJudgeWorkflow`	End-to-end config-driven LLM judge workflow.

API

class nemo_curator.eval.llm_judge.LLMJudgeWorkflow(
    judge_config: str | pathlib.Path,
    input_path: str,
    output_path: str,
    input_format: nemo_curator.eval.llm_judge.workflow.DataFormat = 'jsonl',
    output_format: nemo_curator.eval.llm_judge.workflow.DataFormat = 'jsonl',
    files_per_partition: int | None = None,
    language: str | None = None,
    fasttext_langid_model_path: str | None = None,
    min_langid_score: float = 0.3,
    language_text_field: str = 'raw_text',
    checkpoint_path: str | None = None
)

Dataclass

Bases: WorkflowBase

End-to-end config-driven LLM judge workflow.

Loads a judge config YAML (models, Jinja prompt templates, score rubrics, and execution.stages), starts a Dynamo/vLLM inference server hosting the configured judge models, then runs one Curator pipeline containing: reader -> optional FastText language gate -> one NDD DataDesignerStage (+ its filters) per judge stage -> writer.

checkpoint_path

str | None = None

config

dict[str, object] = field(init=False)

config_path

Path = field(init=False)

fasttext_langid_model_path

str | None = None

files_per_partition

int | None = None

input_format

DataFormat = 'jsonl'

input_path

str

judge_config

str | Path

language

str | None = None

language_text_field

str = 'raw_text'

min_langid_score

float = 0.3

output_format

DataFormat = 'jsonl'

output_path

str

nemo_curator.eval.llm_judge.LLMJudgeWorkflow.__post_init__() -> None

nemo_curator.eval.llm_judge.LLMJudgeWorkflow._build_judge_stages(
    endpoint: str
) -> list[tuple[str, data_designer.config.DataDesignerConfigBuilder, list[data_designer.config.ModelProvider], dict[str, object] | None, int | None, list[dict[str, object]]]]

nemo_curator.eval.llm_judge.LLMJudgeWorkflow.run() -> nemo_curator.pipeline.workflow.WorkflowRunResult

Run the complete LLM judge pipeline.

Returns: WorkflowRunResult

WorkflowRunResult containing the pipeline output tasks and timing metadata.