nat.plugins.langchain.eval.langsmith_evaluator_adapter#

Classes#

`_EvaluatorConvention`	Detected evaluator calling convention.
`LangSmithEvaluatorAdapter`	NAT evaluator adapter that wraps a LangSmith/openevals evaluator callable.

Functions#

_invoke_maybe_sync(→ Any)

Invoke fn with the given arguments, adapting sync callables to async.

Module Contents#

class _EvaluatorConvention#

Bases: enum.StrEnum

Detected evaluator calling convention.

Initialize self. See help(type(self)) for accurate signature.

RUN_EVALUATOR_CLASS = 'run_evaluator_class'#

RUN_EXAMPLE_FUNCTION = 'run_example_function'#

OPENEVALS_FUNCTION = 'openevals_function'#

async _invoke_maybe_sync(fn: collections.abc.Callable[Ellipsis, Any], *args: Any, \*\*kwargs: Any) → Any#

Invoke fn with the given arguments, adapting sync callables to async.

If fn is a coroutine function it is awaited directly. Otherwise it is dispatched to the default executor so that it never blocks the event loop.

The current contextvars context is explicitly copied into the executor thread so that caller-side context managers (e.g., tracing_context(enabled=False)) remain effective.

class LangSmithEvaluatorAdapter( evaluator: Any, convention: str, max_concurrency: int = 4, evaluator_name: str = 'langsmith', extra_fields: dict[str, str] | None = None, score_field: str | None = None, )#

Bases: nat.plugins.eval.evaluator.base_evaluator.BaseEvaluator

NAT evaluator adapter that wraps a LangSmith/openevals evaluator callable.

Adapts various LangSmith evaluator calling conventions into NAT’s BaseEvaluator interface:

RunEvaluator: calls aevaluate_run with synthetic Run/Example objects
(run, example) functions: constructs synthetic Run/Example objects
(inputs, outputs, reference_outputs) functions: passes kwargs directly

All evaluator calls are wrapped in tracing_context(enabled=False) so that LangSmith auto-tracing does not produce unintended traces. NAT’s own observability pipeline (OTEL-based LangSmith exporter) handles tracing separately.

_evaluator#

_extra_fields = None#

_score_field = None#

async evaluate_item( item: nat.data_models.evaluator.EvalInputItem, ) → nat.data_models.evaluator.EvalOutputItem#: Evaluate a single item using the wrapped evaluator.

async _call_run_evaluator( item: nat.data_models.evaluator.EvalInputItem, ) → Any#: Call a RunEvaluator subclass instance via aevaluate_run.

async _call_run_example_function( item: nat.data_models.evaluator.EvalInputItem, ) → Any#: Call a function with (run, example) signature.

async _call_openevals_function( item: nat.data_models.evaluator.EvalInputItem, ) → Any#: Call a function with (inputs, outputs, reference_outputs) signature.