nat.plugins.langchain.eval.langsmith_custom_evaluator

Attributes

logger

Classes

LangSmithCustomEvaluatorConfig

Import any LangSmith-compatible evaluator by Python dotted path.

Functions

_import_evaluator(→ Any)	Import an evaluator from a Python dotted path.
_detect_convention(→ str)	Auto-detect which LangSmith evaluator convention is being used.
register_langsmith_custom_evaluator(config, builder)	Register a custom LangSmith evaluator with NAT.

Module Contents

logger

_import_evaluator(dotted_path: str) → Any

Import an evaluator from a Python dotted path.

Supports both module-level callables and class references: - 'my_package.evaluators.my_function' -> imports and returns the function - 'my_package.evaluators.MyClass' -> imports and instantiates the class

Args:

dotted_path: Full Python dotted path to the evaluator.

Returns:

The imported evaluator (callable or instance).

Raises:

ImportError: If the module cannot be imported. AttributeError: If the attribute cannot be found in the module.

_detect_convention(evaluator: Any) → str

Auto-detect which LangSmith evaluator convention is being used.

Inspects the evaluator to determine if it’s a RunEvaluator subclass, a function with (run, example) signature, or a function with (inputs, outputs, reference_outputs) signature.

Args:

evaluator: The evaluator callable or instance.

Returns:

One of 'run_evaluator_class', 'run_example_function', or 'openevals_function'.

class LangSmithCustomEvaluatorConfig

Bases: nat.data_models.evaluator.EvaluatorBaseConfig, nat.plugins.langchain.eval.langsmith_evaluator.LangSmithExtraFieldsMixin

Import any LangSmith-compatible evaluator by Python dotted path.

Supports RunEvaluator subclasses, (run, example) functions, and (inputs, outputs, reference_outputs) functions. The calling convention is auto-detected at registration time.

For built-in openevals evaluators, prefer _type: langsmith with a short name instead.

evaluator: str = None

async register_langsmith_custom_evaluator(

config: LangSmithCustomEvaluatorConfig,

builder: nat.builder.builder.EvalBuilder,

)

Register a custom LangSmith evaluator with NAT.