aiq.eval.swe_bench_evaluator.register

Classes

SweBenchEvaluatorConfig

Code patch evaluation for SWE Bench problems.

Functions

register_swe_bench_evaluator(config, builder)

Module Contents

class SweBenchEvaluatorConfig(/, **data: Any)

Bases: aiq.data_models.evaluator.EvaluatorBaseConfig

Code patch evaluation for SWE Bench problems.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

run_id: str = None

async register_swe_bench_evaluator(

config: SweBenchEvaluatorConfig,

builder: aiq.builder.builder.EvalBuilder,

)