Basic Audit Target#

When you run an audit job in NVIDIA NeMo Auditor, you create a separate audit target and audit configuration for the job.

The target specifies the model name, model type, and free-form key-value pairs for model-specific inference options.

Schema for Audit Targets#

Field

Description

Default Value

name

Specifies the name of the audit target. Names must be unique within a namespace. The maximum length is 250 characters.

None (Required)

namespace

Specifies the namespace for the audit target. The maximum length is 250 characters.

None (Required)

options

Specifies a dictionary of model-specific inference options. Specify the model_type as the top-level key and then the key-value pairs.

The following sample JSON is valid when model_type is nim:

"options": {
  "nim": {
    "context_len": null,
    "extra_params": {},
    "frequency_penalty": 0.0,
    "max_tokens": 150,
    "presence_penalty": 0.0,
    "retry_json": true,
    "seed": null,
    "skip_seq_end": null,
    "skip_seq_start": null,
    "stop": ["#", ";"],
    "suppressed_params": [
       "frequency_penalty",
       "n",
       "presence_penalty",
       "timeout"
     ],
     "temperature": 0.1,
     "top_k": 0,
     "top_p": 0.7,
     "uri": "https://integrate.api.nvidia.com/v1/",
     "vary_seed_each_call": true,
     "vary_temp_each_call": true
  }
}

To access models from NeMo NIM Proxy, set the service name as the uri:

"generator_config": {
  "nim": {
    "uri": "http://nemo-nim-proxy:8000/v1/"
  }
}

Refer to the DEFAULT_PARAMS for the generator class:

model_name

Specifies the model to audit, such as o3-mini or meta/llama-3.1-8b-instruct.

None (Required)

model_type

Specifies the type of model to audit. These correspond to a garak generator.

The microservice supports a subset of the generators that garak supports. Specify one of the following generators:

  • openai

  • nim

  • rest

None (Required)

The following target references the NVIDIA Llama 3.1 Nemotron Nano V1 8B model from a locally-accessible NIM microservice. The service name for the LLM container is llm.

Set AUDITOR_BASE_URL to specify the service:

$ export AUDITOR_BASE_URL=http://localhost:5000

import os
from nemo_microservices import NeMoMicroservices

client = NeMoMicroservices(base_url=os.getenv("AUDITOR_BASE_URL"))
    
target = client.beta.audit.targets.create(
    namespace="default",
    name="demo-basic-target",
    type="nim.NVOpenAIChat",
    model="deepseek-ai/deepseek-r1-distill-llama-8b",
    options={
        "nim": {
            "skip_seq_start": "<think>",
            "skip_seq_end": "</think>",
            "max_tokens": 3200,
            "uri": "http://llm:8000/v1/"
        }
    }
)
print(target)

curl -X POST "${AUDITOR_BASE_URL}/v1beta1/audit/targets" \
  -H "Accept: application/json" \
  -H "Content-Type: application/json" \
  -d '{
    "namespace": "default",
    "name": "demo-basic-target",
    "type": "nim.NVOpenAIChat",
    "model": "deepseek-ai/deepseek-r1-distill-llama-8b",
    "options": {
      "nim": {
          "skip_seq_start": "<think>",
          "skip_seq_end": "</think>",
          "max_tokens": 3200,
          "uri": "http://llm:8000/v1/"
      }
    }
  }' | jq