Locally-Deployed NIM Microservice Target

The following target references the NVIDIA Llama 3.1 Nemotron Nano V1 8B model from a locally-accessible NIM microservice. The service name for the LLM container is llm.

Refer to garak.generators.nim.NVOpenAIChat for the parameters to specify in the options.nim field. The options override the default values from the DEFAULT_PARAMS in the API reference.

Important

Export the NIM_API_KEY environment variable with your API key or any value when you start the microservice container. The environment variable must be set even if it is not used to access build.nvidia.com.

Python SDK

import os
from nemo_microservices import NeMoMicroservices

client = NeMoMicroservices(base_url=os.getenv("AUDITOR_BASE_URL"))
    
target = client.beta.audit.targets.create(
    namespace="default",
    name="demo-basic-target",
    type="nim.NVOpenAIChat",
    model="deepseek-ai/deepseek-r1-distill-llama-8b",
    options={
        "nim": {
            "skip_seq_start": "<think>",
            "skip_seq_end": "</think>",
            "max_tokens": 3200,
            "uri": "http://llm:8000/v1/"
        }
    }
)
print(target)

cURL

curl -X POST "${AUDITOR_BASE_URL}/v1beta1/audit/targets" \
  -H "Accept: application/json" \
  -H "Content-Type: application/json" \
  -d '{
    "namespace": "default",
    "name": "demo-basic-target",
    "type": "nim.NVOpenAIChat",
    "model": "deepseek-ai/deepseek-r1-distill-llama-8b",
    "options": {
      "nim": {
          "skip_seq_start": "<think>",
          "skip_seq_end": "</think>",
          "max_tokens": 3200,
          "uri": "http://llm:8000/v1/"
      }
    }
  }' | jq

Example Output

Python SDK

AuditTarget(model='deepseek-ai/deepseek-r1-distill-llama-8b',
type='nim.NVOpenAIChat', id='audit_target-2zZ8vpxU7c7xSN4AyDJqQZ',
created_at=datetime.datetime(2025, 7, 24, 17, 20, 31, 764909),
custom_fields={}, description=None, name='demo-basic-target',
namespace='default', options={'nim': {'skip_seq_start': '<think>',
'skip_seq_end': '</think>', 'max_tokens': 3200, 'uri':
'http://llm:8000/v1/'}}, ownership=None, project=None,
schema_version='1.0', type_prefix=None, updated_at=datetime.datetime(2025,
7, 24, 17, 20, 31, 764913))

cURL

{
  "schema_version": "1.0",
  "id": "audit_target-2brsLXr44iGWGH3Sdvi1wF",
  "description": null,
  "type_prefix": null,
  "namespace": "default",
  "project": null,
  "created_at": "2025-07-24T16:50:07.930815",
  "updated_at": "2025-07-24T16:50:07.930818",
  "custom_fields": {},
  "ownership": null,
  "name": "demo-basic-target",
  "type": "nim.NVOpenAIChat",
  "model": "deepseek-ai/deepseek-r1-distill-llama-8b",
  "options": {
    "nim": {
      "skip_seq_start": "<think>",
      "skip_seq_end": "</think>",
      "max_tokens": 3200,
      "uri": "http://llm:8000/v1/"
    }
  }
}