nat.plugins.langchain.eval.langsmith_evaluator#

Attributes#

logger

Classes#

`LangSmithExtraFieldsMixin`	Mixin for extra fields on the LangSmith evaluator config.
`LangSmithEvaluatorConfig`	Built-in openevals evaluator selected by short name.

Functions#

`_get_registry`(→ dict[str, ...)	Return the evaluator registry, importing openevals lazily.
`_resolve_evaluator`(...)	Resolve a short evaluator name to its openevals callable.
`register_langsmith_evaluator`(config, builder)	Register a built-in openevals evaluator with NAT.

Module Contents#

logger#

_get_registry() → dict[str, collections.abc.Callable[Ellipsis, Any]]#

Return the evaluator registry, importing openevals lazily.

Keeps openevals out of the module-level import chain while providing a single source of truth for known evaluator names and their callables.

Async variants are used to align with NAT’s async-first design. The adapter (LangSmithEvaluatorAdapter) awaits async callables directly via _invoke_maybe_sync, avoiding unnecessary thread-pool dispatch.

_resolve_evaluator( name: str, ) → collections.abc.Callable[Ellipsis, Any]#

Resolve a short evaluator name to its openevals callable.

The model validator on LangSmithEvaluatorConfig already ensures name is valid, so this is a direct lookup.

Args:: name: Short evaluator name (e.g., 'exact_match', 'levenshtein_distance').
Returns:: The resolved evaluator callable.

class LangSmithExtraFieldsMixin(/, **data: Any)#

Bases: pydantic.BaseModel

Mixin for extra fields on the LangSmith evaluator config.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

extra_fields: dict[str, str] | None = None#

class LangSmithEvaluatorConfig(/, **data: Any)#

Bases: nat.data_models.evaluator.EvaluatorBaseConfig, LangSmithExtraFieldsMixin

Built-in openevals evaluator selected by short name.

Resolves evaluator names (e.g., 'exact_match', 'levenshtein_distance') from the openevals package automatically. For custom user-defined evaluators, use _type: langsmith_custom instead.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

evaluator: str = None#

_validate_evaluator_name() → LangSmithEvaluatorConfig#: Validate that the evaluator name exists in the registry.

async register_langsmith_evaluator( config: LangSmithEvaluatorConfig, builder: nat.builder.builder.EvalBuilder, )#: Register a built-in openevals evaluator with NAT.