nat.plugins.profiler.runtime_evaluator.atif_evaluate#

ATIF-native runtime evaluators for the profiler package.

Classes#

`AverageLLMLatencyAtifEvaluator`	ATIF-native mean latency between LLM start and end for agent steps with metrics.
`AverageWorkflowRuntimeAtifEvaluator`	ATIF-native workflow runtime per item: max(step.timestamp) - min(step.timestamp) across all steps.
`AverageNumberOfLLMCallsAtifEvaluator`	ATIF-native count of LLM calls per item: agent steps with metrics.
`AverageTokensPerLLMEndAtifEvaluator`	ATIF-native average total tokens per LLM call: (prompt_tokens + completion_tokens) from step.metrics.

Functions#

`_iso_to_epoch`(→ float \| None)	Convert ISO 8601 timestamp to epoch seconds, or None if invalid.
`_step_llm_latency`(→ float \| None)	Return the LLM start-to-end latency (seconds) for one ATIF agent step.

Module Contents#

_iso_to_epoch(ts: str | None) → float | None#: Convert ISO 8601 timestamp to epoch seconds, or None if invalid.

_step_llm_latency(step: nat.atif.Step) → float | None#

Return the LLM start-to-end latency (seconds) for one ATIF agent step.

Prefers the converter-written step.extra["invocation"] timing (start_timestamp / end_timestamp in epoch seconds), which is what the ATOF-to-ATIF converter actually emits. Falls back to the legacy step.extra["span_event_timestamp"] (ISO start) paired with step.timestamp (ISO end) for producers that emit that shape instead.

Returns None when neither timing source is present, or when the invocation timestamps are not finite, so the caller can skip the step rather than record a spurious zero.

class AverageLLMLatencyAtifEvaluator(max_concurrency: int = 8)#

Bases: nat.plugins.eval.evaluator.atif_base_evaluator.AtifBaseEvaluator

ATIF-native mean latency between LLM start and end for agent steps with metrics.

Per step, prefers the step.extra["invocation"] start/end timestamps written by the ATOF-to-ATIF converter, and falls back to step.extra["span_event_timestamp"] (start) paired with step.timestamp (end). A step is skipped only when neither timing source is available (see NEP-008 for ATIF profiling metadata).

async evaluate_atif_item( sample: nat.plugins.eval.evaluator.atif_evaluator.AtifEvalSample, ) → nat.plugins.eval.data_models.evaluator_io.EvalOutputItem#: Evaluate one ATIF sample and return a single output item.

class AverageWorkflowRuntimeAtifEvaluator(max_concurrency: int = 8)#

Bases: nat.plugins.eval.evaluator.atif_base_evaluator.AtifBaseEvaluator

ATIF-native workflow runtime per item: max(step.timestamp) - min(step.timestamp) across all steps.

async evaluate_atif_item( sample: nat.plugins.eval.evaluator.atif_evaluator.AtifEvalSample, ) → nat.plugins.eval.data_models.evaluator_io.EvalOutputItem#: Evaluate one ATIF sample and return a single output item.

class AverageNumberOfLLMCallsAtifEvaluator(max_concurrency: int = 8)#

Bases: nat.plugins.eval.evaluator.atif_base_evaluator.AtifBaseEvaluator

ATIF-native count of LLM calls per item: agent steps with metrics.

async evaluate_atif_item( sample: nat.plugins.eval.evaluator.atif_evaluator.AtifEvalSample, ) → nat.plugins.eval.data_models.evaluator_io.EvalOutputItem#: Evaluate one ATIF sample and return a single output item.

class AverageTokensPerLLMEndAtifEvaluator(max_concurrency: int = 8)#

Bases: nat.plugins.eval.evaluator.atif_base_evaluator.AtifBaseEvaluator

ATIF-native average total tokens per LLM call: (prompt_tokens + completion_tokens) from step.metrics.

async evaluate_atif_item( sample: nat.plugins.eval.evaluator.atif_evaluator.AtifEvalSample, ) → nat.plugins.eval.data_models.evaluator_io.EvalOutputItem#: Evaluate one ATIF sample and return a single output item.