Model Configuration
Online evaluations use Model objects for model endpoints. A model can be the evaluation target that produces outputs, or it can be part of a judge-style metric such as LLM-as-a-Judge, RAG, or agentic metrics.
The Evals plugin SDK uses inline model objects from nhx_evals_sdk. Pass the model either as target=... or as a field on the metric class that needs a judge or embeddings model.
Initialize the SDK
These examples use the default workspace. If your current CLI context uses a different workspace, run nemo config set --workspace default first. To check, run nemo config view.
Inline Model
Define the endpoint URL and model name directly:
Set NVIDIA_BASE_URL to override the full OpenAI-compatible API base URL (including /v1),
for example https://inference-api.nvidia.com/v1. The default remains https://integrate.api.nvidia.com/v1.
Model API Authentication
api_key_secret is an optional property on the Model object. Omit it when the endpoint does not require API-key authentication.
For nhx_evals_sdk.Evaluator runs, the SecretRef root names an environment variable available to the local Python process. For example, api_key_secret=SecretRef(root="NVIDIA_API_KEY") reads os.environ["NVIDIA_API_KEY"].
For remote evaluator.submit(...) jobs, the SecretRef root must name a NeMo Helix secret in the target workspace. Create the secret before submitting the job:
Model as the Evaluation Target
Use target=model when the evaluator should call the model to generate the sample output before scoring.
Model on a Judge Metric
Use a model field on the metric when the metric itself calls an LLM to score existing outputs.
Runtime Parameters
Use RunConfigOnlineModel for model-target evaluations:
Use plain RunConfig for offline evaluations where the dataset already contains the output to score.
Model References
You can supply the evaluation target two ways. Which one is valid depends on whether you submit the evaluation as a durable platform job or run it locally with nhx_evals_sdk.Evaluator.
Inline Model
An inline Model carries the resolved endpoint details. Always pass an inline Model as the target (or as a judge/embeddings field on the metric). If your deployment stores platform model entities, resolve the entity into endpoint details before constructing the Model:
ModelRef (supported by evaluator.submit(...))
Durable remote evaluator.submit(...) jobs additionally accept a ModelRef target. A ModelRef names a model the Inference Gateway serves, a platform model entity or a virtual model (workspace/name), and is resolved by the evaluator backend when the job runs, so you do not have to resolve the endpoint yourself. Use this for platform-managed model routing. A ModelRef target generates outputs online, so it requires an online run config (RunConfigOnlineModel):
evaluator.submit(...) accepts either a Model or a ModelRef. nhx_evals_sdk.Evaluator resolves no platform entities, so pass it an inline Model. See the Define and Run Custom Python Metrics tutorial for an end-to-end ModelRef + FilesetRef submit example.
For evaluating agentic systems, use an Agent request target instead of a Model. See Agent Configuration.
Images in the Prompt
A vision-language model reads images from image_url parts in a chat message. A request through the Inference Gateway can give the image in three forms:
data:image/png;base64,...: the image bytes, inline in the request.http(s)://...: the model server downloads the image itself.fileset://<workspace>/<fileset>#<path>: the gateway downloads the file from the fileset with the caller’s permissions, and inlines it as adata:URL. PNG, JPEG, GIF, or WebP, up to 25 MiB.
Dataset rows usually hold image paths relative to the fileset they were uploaded to, which is what training reads. Write the fileset:// reference in the prompt template, and leave the rows as they are. This example scores a receipt-reading model on the CORD-v2 validation split. Each row’s user turn is one image part followed by one text part, and its assistant turn is the expected answer:
For the first row, the request carries fileset://default/cord-v2#validation/images/00000.png. The job downloads only the JSONL, and the gateway fetches each image as its request arrives, so saved results hold the short reference rather than the image bytes.
- The template’s indices follow the row shown in the comment:
messages[0]is the user turn,content[0]is its image part, andcontent[1]is its text part. Rows with a different shape need different indices. - The request must go through the Inference Gateway, as a
ModelReftarget’s requests do. A model server called directly reads onlydata:andhttp(s)URLs. - The template names the fileset, so a copy of the dataset under another name needs the template changed.
- When rows already hold
data:,http(s), orfileset://URLs,{"messages": "{{ item.messages[:-1] }}"}sends their image parts unchanged.