nemo_evaluator.adapters

Interceptors and PostEvalHooks are important part of NeMo Evaluator SDK. They expand functionality of each harness, providing a standardized way of enabling features in your evaluation runs. Behind each interceptor and post-eval-hook stands a specific class that implements its logic. However, these classes are referenced only to provide their configuration options which are reflected in Params of each class and to point from which classes one should inherit.

From the usage perspective, one should always use configuration classes (see Configuration) to add them to evaluations. No interceptor should be directly instantiated.

Interceptors are defined in a chain. They go under target.api_endpoint.adapter_config and can be defined as follow:

adapter_config = AdapterConfig(
    interceptors=[
        InterceptorConfig(
            name="system_message",
            enabled=True,
            config={"system_message": "You are a helpful assistant."}
        ),
        InterceptorConfig(name="request_logging", enabled=True),
        InterceptorConfig(
            name="caching",
            enabled=True,
            config={"cache_dir": "./cache", "reuse_cached_responses": True}
        ),
        InterceptorConfig(name="reasoning", enabled=True),
        InterceptorConfig(name="endpoint")
    ]
)

Configuration

DiscoveryConfig	Configuration for discovering 3rd party modules and directories
InterceptorConfig	Configuration for a single interceptor
PostEvalHookConfig	Configuration for a single post-evaluation hook
AdapterConfig	Adapter configuration with registry-based interceptor support

Interceptors

CachingInterceptor	Caching interceptor is special in the sense that it intercepts both requests and responses.
EndpointInterceptor	Required interceptor that handles the actual API communication.
PayloadParamsModifierInterceptor	Adapter for modifying request payload by removing, adding, and renaming parameters
ProgressTrackingInterceptor	Progress tracking via external webhook.
RaiseClientErrorInterceptor	Adapter for handling non-retryable client error to raise an exception instead of continuing the benchmark.
RequestLoggingInterceptor	Logs incoming requests.
ResponseLoggingInterceptor	Logs responses.
ResponseReasoningInterceptor	Processes reasoning tokens from response.
ResponseStatsInterceptor	Collects aggregated statistics from API responses for metrics collection.
SystemMessageInterceptor	Adds or replaces system message in requests.

PostEvalHooks

PostEvalReportHook

Post-evaluation hook that generates reports from cached requests and responses.

Interfaces

RequestInterceptor	Interface for providing interception of requests.
ResponseInterceptor	Interface for providing interception of responses.
RequestToResponseInterceptor	Interface for interceptors that can either continue the request chain or return a response.
PostEvalHook	Interface for post-evaluation hooks that run after the evaluation completes.