nat.profiler.parameter_optimization.optimizer_callbacks#

Attributes#

logger

Classes#

`TrialResult`
`OptimizerCallback`	Base class for protocol classes.
`OptimizerCallbackManager`

Module Contents#

logger#

class TrialResult#

trial_number: int#

parameters: dict[str, Any]#

metric_scores: dict[str, float]#

is_best: bool#

rep_scores: list[list[float]] | None = None#

prompts: dict[str, str] | None = None#

prompt_formats: dict[str, str] | None = None#

eval_result: Any | None = None#

class OptimizerCallback#

Bases: Protocol

Base class for protocol classes.

Protocol classes are defined as:

class Proto(Protocol):
    def meth(self) -> int:
        ...

Such classes are primarily used with static type checkers that recognize structural subtyping (static duck-typing).

For example:

class C:
    def meth(self) -> int:
        return 0

def func(x: Proto) -> int:
    return x.meth()

func(C())  # Passes static type check

See PEP 544 for details. Protocol classes decorated with @typing.runtime_checkable act as simple-minded runtime protocols that check only the presence of given attributes, ignoring their type signatures. Protocol classes can be generic, they are defined as:

class GenProto[T](Protocol):
    def meth(self) -> T:
        ...

pre_create_experiment( dataset_items: list[nat.eval.evaluator.evaluator_model.EvalInputItem], ) → None#

on_trial_end(result: TrialResult) → None#

on_study_end(*, best_trial: TrialResult, total_trials: int) → None#

class OptimizerCallbackManager#

_callbacks: list[OptimizerCallback] = []#

register(callback: OptimizerCallback) → None#

property has_callbacks: bool#

set_prompt_param_names(names: list[str]) → None#

pre_create_experiment( dataset_items: list[nat.eval.evaluator.evaluator_model.EvalInputItem], ) → None#

on_trial_end(result: TrialResult) → None#

get_trial_project_name(trial_number: int) → str | None#: Get a trial-specific OTEL project name from the first callback that supports it.

on_study_end(*, best_trial: TrialResult, total_trials: int) → None#