Manage Audit Jobs#

After you create an audit target and an audit configuration, you are ready to run an audit job.

Prerequisites#

Create an Audit Job#

Tip

Before proceeding, ensure NeMo Auditor is running and accessible. If you are running NeMo Auditor in a Kubernetes cluster, you can use port-forwarding to access the service locally: kubectl port-forward svc/auditor 5000:5000 -n <namespace>

For information about the fields, refer to Audit Job Schema.

  1. Create the job with the basic target and basic config:

    import os
from nemo_microservices import NeMoMicroservices

client = NeMoMicroservices(base_url=os.getenv("AUDITOR_BASE_URL"))

job = client.beta.audit.jobs.create(
    name="demo-basic-job",
    project="demo",
    spec={
        "config": "default/default",
        "target": "default/demo-basic-target"
    },
)
print(job.id)
print(job.model_dump_json(indent=2))

    curl -X POST "${AUDITOR_BASE_URL}/v1beta1/audit/jobs" \
  -H "Accept: application/json" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "demo-basic-job",
    "project": "demo",
    "spec": {
        "config": "default/default",
        "target": "default/demo-basic-target"
    }
}' | jq

    Example Output

    job-pcanyxxhidh55rvhdwnh9y

{
  "name": "demo-basic-job",
  "spec": {
    "config": "default/demo-basic-config",
    "target": "default/demo-local-llm-target"
  },
  "id": "job-pcanyxxhidh55rvhdwnh9y",
  "created_at": "2025-09-24T14:22:01.853151",
  "custom_fields": null,
  "description": null,
  "error_details": null,
  "namespace": "default",
  "ownership": null,
  "project": "demo",
  "status": "created",
  "status_details": {},
  "updated_at": "2025-09-24T14:22:01.853157"
}

    {
  "id": "job-hdwghdv2tw7ji5tm75toh1",
  "name": "demo-basic-job",
  "description": null,
  "project": "demo",
  "namespace": "default",
  "created_at": "2025-10-22T20:15:06.879649",
  "updated_at": "2025-10-22T20:15:06.879654",
  "spec": {
    "config": "default/demo-basic-config",
    "target": "default/demo-nemo-platform-target",
    "task_options": {
      "max_probe_retries": 0,
      "fail_job_on_retries_exhausted": true
    }
  },
  "status": "created",
  "status_details": {},
  "error_details": null,
  "ownership": null,
  "custom_fields": null
}

After you create the job, check the status to ensure it becomes active.

Get Audit Job Status#

A Job can report the following statuses:

created

The job exists but has not yet been submitted to a job executor. This is the initial state for all new jobs.

pending

The job is waiting for active execution.

active

The job is currently executing its tasks.

error

The job terminated due to an unrecoverable error and cannot be retried.

cancelling

The Jobs microservice is attempting to terminate the job gracefully.

cancelled

The job is terminated as a result of a cancellation request.

pausing

The Jobs microservice is attempting to pause the job.

paused

The job is paused as a result of a pause request.

resuming:

The paused job is resuming to the active state as a result of a resume request.

completed

The job finished all its steps successfully without errors.

Send a GET request to the /v1beta1/audit/jobs/{id}/status endpoint.

import os
from nemo_microservices import NeMoMicroservices

client = NeMoMicroservices(base_url=os.getenv("AUDITOR_BASE_URL"))

status = client.beta.audit.jobs.get_status(job_id)
print(status.model_dump_json(indent=2))

curl "${AUDITOR_BASE_URL}/v1beta1/audit/jobs/${JOB_ID}/status" \
  -H "Accept: application/json" | jq

Example Output

{
  "error_details": null,
  "job_id": "job-n1p4dtay92jlag7b2bq8sp",
  "status": "completed",
  "status_details": {
    "progress": {
      "probes_total": 2,
      "probes_complete": 2,
      "probes_failed": 0
    },
    "runtime_probe_status": {
      "probe_name": "dan.AutoDANCached",
      "retry": 0,
      "max_probe_retries": 0
    }
  },
  "steps": [
    {
      "error_details": {},
      "name": "audit",
      "status": "completed",
      "status_details": {},
      "tasks": [
        {
          "id": "ba5b04ef-4c18-4301-a993-49c90f6376e2",
          "error_details": {},
          "error_stack": null,
          "status": "completed",
          "status_details": {}
        }
      ]
    }
  ]
}

{
  "job_id": "job-hdwghdv2tw7ji5tm75toh1",
  "status": "completed",
  "status_details": {
    "progress": {
      "probes_total": 2,
      "probes_complete": 2,
      "probes_failed": 0
    },
    "runtime_probe_status": {
      "probe_name": "dan.AutoDANCached",
      "retry": 0,
      "max_probe_retries": 0
    }
  },
  "error_details": null,
  "steps": [
    {
      "name": "audit",
      "status": "completed",
      "status_details": {},
      "error_details": {},
      "tasks": [
        {
          "id": "cbb6ecac-d26e-46ef-9d31-13fea8865c93",
          "status": "completed",
          "status_details": {},
          "error_details": {},
          "error_stack": null
        }
      ]
    }
  ]
}

After you confirm a job is active, view the audit job log to check for progress.

Get an Audit Job Log#

You can retrieve a log of the job progress. The log is produced by garak for each probe.

  • Send a GET request to the /v1beta1/audit/jobs/{id}/logs endpoint.

    client = NeMoMicroservices(base_url=os.getenv("AUDITOR_BASE_URL"))
logs = client.beta.audit.jobs.get_logs(job_id)
print("".join(log.message for log in logs.data[-10:]))

    curl "${AUDITOR_BASE_URL}/v1beta1/audit/jobs/${JOB_ID}/logs?limit=99999" \
  -H "Accept: text/plain" | jq '.data[].message'
    Partial Job Log
    2025-10-23 18:09:13,219  DEBUG  langauge provision service: en,en
2025-10-23 18:09:13,239  INFO  detector init: <garak.detectors.base.TriggerListDetector object at 0x7fb0aa5f74d0>
2025-10-23 18:09:13,239  DEBUG  harness: probe start for garak.probes.goodside.Tag
2025-10-23 18:09:13,239  DEBUG  probe execute: <garak.probes.goodside.Tag object at 0x7fb0aa648110>
2025-10-23 18:09:15,359  DEBUG  receive_response_headers.complete return_value=(b'HTTP/1.1', 200, b'OK', [(b'Content-Length', b'674'), (b'Content-Type', b'application/json'), (b'Date', b'Thu, 23 Oct 2025 18:09:13 GMT'), (b'Server', b'uvicorn'), (b'X-Request-Id', b'nemo-nim-proxy-8d4b7b66c-qrcpg/M7LDfmH3ma-044838')])
2025-10-23 18:09:15,360  INFO  HTTP Request: POST http://nemo-nim-proxy:8000/v1/chat/completions "HTTP/1.1 200 OK"
2025-10-23 18:09:15,360  DEBUG  receive_response_body.started request=<Request [b'POST']>
2025-10-23 18:09:15,360  DEBUG  receive_response_body.complete
2025-10-23 18:09:15,361  DEBUG  response_closed.started
2025-10-23 18:09:15,361  DEBUG  response_closed.complete

List all Jobs#

The /v1beta1/audit/jobs endpoint lists all jobs since the microservice was last started.

jobs = client.beta.audit.jobs.list()
print("\n".join([job.model_dump_json(indent=2) for job in jobs]))

curl -X GET "${AUDITOR_BASE_URL}/v1beta1/audit/jobs" \
  -H "Accept: application/json" | jq

Example Output

{
  "name": "demo-inference-count-job",
  "spec": {
    "config": "default/demo-config-using-tags",
    "target": "default/demo-blank-target",
    "task_options": {
      "fail_job_on_retries_exhausted": true,
      "max_probe_retries": 0
    }
  },
  "id": "job-dgtt2kilayqivjizj1cncv",
  "created_at": "2025-10-23T18:10:55.765190",
  "custom_fields": null,
  "description": null,
  "error_details": null,
  "namespace": "default",
  "ownership": null,
  "project": "demo",
  "status": "completed",
  "status_details": {
    "progress": {
      "probes_total": 21,
      "probes_complete": 21,
      "probes_failed": 0
    },
    "runtime_probe_status": {
      "probe_name": "leakreplay.GuardianComplete",
      "retry": 0,
      "max_probe_retries": 0
    }
  },
  "updated_at": "2025-10-23T18:10:55.765193"
}
{
  "name": "demo-basic-job",
  "spec": {
    "config": "default/demo-basic-config",
    "target": "default/demo-nemo-platform-target",
    "task_options": {
      "fail_job_on_retries_exhausted": true,
      "max_probe_retries": 0
    }
  },
  "id": "job-n1p4dtay92jlag7b2bq8sp",
  "created_at": "2025-10-23T18:08:53.265465",
  "custom_fields": null,
  "description": null,
  "error_details": null,
  "namespace": "default",
  "ownership": null,
  "project": "demo",
  "status": "completed",
  "status_details": {
    "progress": {
      "probes_total": 2,
      "probes_complete": 2,
      "probes_failed": 0
    },
    "runtime_probe_status": {
      "probe_name": "dan.AutoDANCached",
      "retry": 0,
      "max_probe_retries": 0
    }
  },
  "updated_at": "2025-10-23T18:08:53.265467"
}
{
  "name": "demo-inference-count-job",
  "spec": {
    "config": "default/demo-config-using-tags",
    "target": "default/demo-blank-target",
    "task_options": {
      "fail_job_on_retries_exhausted": true,
      "max_probe_retries": 0
    }
  },
  "id": "job-hna5arbzwjnwfnqxtatt6q",
  "created_at": "2025-10-23T18:03:14.987305",
  "custom_fields": null,
  "description": null,
  "error_details": null,
  "namespace": "default",
  "ownership": null,
  "project": "demo",
  "status": "completed",
  "status_details": {
    "progress": {
      "probes_total": 21,
      "probes_complete": 21,
      "probes_failed": 0
    },
    "runtime_probe_status": {
      "probe_name": "leakreplay.GuardianComplete",
      "retry": 0,
      "max_probe_retries": 0
    }
  },
  "updated_at": "2025-10-23T18:03:14.987309"
}
{
  "name": "demo-basic-job",
  "spec": {
    "config": "default/demo-basic-config",
    "target": "default/demo-nemo-platform-target",
    "task_options": {
      "fail_job_on_retries_exhausted": true,
      "max_probe_retries": 0
    }
  },
  "id": "job-rtrysr6badxmygt1akpbyb",
  "created_at": "2025-10-23T18:00:12.197151",
  "custom_fields": null,
  "description": null,
  "error_details": null,
  "namespace": "default",
  "ownership": null,
  "project": "demo",
  "status": "completed",
  "status_details": {
    "progress": {
      "probes_total": 2,
      "probes_complete": 2,
      "probes_failed": 0
    },
    "runtime_probe_status": {
      "probe_name": "dan.AutoDANCached",
      "retry": 0,
      "max_probe_retries": 0
    }
  },
  "updated_at": "2025-10-23T18:00:12.197156"
}
{
  "name": "demo-basic-job",
  "spec": {
    "config": "default/demo-basic-config",
    "target": "default/demo-nemo-platform-target",
    "task_options": {
      "fail_job_on_retries_exhausted": true,
      "max_probe_retries": 0
    }
  },
  "id": "job-qgbsr8ers5l6ap6c2yqsc",
  "created_at": "2025-10-23T17:45:58.600240",
  "custom_fields": null,
  "description": null,
  "error_details": null,
  "namespace": "default",
  "ownership": null,
  "project": "demo",
  "status": "completed",
  "status_details": {
    "progress": {
      "probes_total": 2,
      "probes_complete": 2,
      "probes_failed": 0
    },
    "runtime_probe_status": {
      "probe_name": "dan.AutoDANCached",
      "retry": 0,
      "max_probe_retries": 0
    }
  },
  "updated_at": "2025-10-23T17:45:58.600244"
}
{
  "name": "demo-inference-count-job",
  "spec": {
    "config": "default/demo-config-using-tags",
    "target": "default/demo-blank-target",
    "task_options": {
      "fail_job_on_retries_exhausted": true,
      "max_probe_retries": 0
    }
  },
  "id": "job-ubxt82qswyg8r8dfrzlhoy",
  "created_at": "2025-10-22T20:18:09.637972",
  "custom_fields": null,
  "description": null,
  "error_details": null,
  "namespace": "default",
  "ownership": null,
  "project": "demo",
  "status": "completed",
  "status_details": {
    "progress": {
      "probes_total": 21,
      "probes_complete": 21,
      "probes_failed": 0
    },
    "runtime_probe_status": {
      "probe_name": "leakreplay.GuardianComplete",
      "retry": 0,
      "max_probe_retries": 0
    }
  },
  "updated_at": "2025-10-22T20:18:09.637977"
}
{
  "name": "demo-basic-job",
  "spec": {
    "config": "default/demo-basic-config",
    "target": "default/demo-nemo-platform-target",
    "task_options": {
      "fail_job_on_retries_exhausted": true,
      "max_probe_retries": 0
    }
  },
  "id": "job-hdwghdv2tw7ji5tm75toh1",
  "created_at": "2025-10-22T20:15:06.879649",
  "custom_fields": null,
  "description": null,
  "error_details": null,
  "namespace": "default",
  "ownership": null,
  "project": "demo",
  "status": "completed",
  "status_details": {
    "progress": {
      "probes_total": 2,
      "probes_complete": 2,
      "probes_failed": 0
    },
    "runtime_probe_status": {
      "probe_name": "dan.AutoDANCached",
      "retry": 0,
      "max_probe_retries": 0
    }
  },
  "updated_at": "2025-10-22T20:15:06.879654"
}

{
  "object": "list",
  "data": [
    {
      "id": "job-ubxt82qswyg8r8dfrzlhoy",
      "name": "demo-inference-count-job",
      "project": "demo",
      "namespace": "default",
      "created_at": "2025-10-22T20:18:09.637972",
      "updated_at": "2025-10-22T20:18:09.637977",
      "spec": {
        "config": "default/demo-config-using-tags",
        "target": "default/demo-blank-target",
        "task_options": {
          "max_probe_retries": 0,
          "fail_job_on_retries_exhausted": true
        }
      },
      "status": "completed",
      "status_details": {
        "progress": {
          "probes_total": 21,
          "probes_complete": 21,
          "probes_failed": 0
        },
        "runtime_probe_status": {
          "probe_name": "leakreplay.GuardianComplete",
          "retry": 0,
          "max_probe_retries": 0
        }
      }
    },
    {
      "id": "job-hdwghdv2tw7ji5tm75toh1",
      "name": "demo-basic-job",
      "project": "demo",
      "namespace": "default",
      "created_at": "2025-10-22T20:15:06.879649",
      "updated_at": "2025-10-22T20:15:06.879654",
      "spec": {
        "config": "default/demo-basic-config",
        "target": "default/demo-nemo-platform-target",
        "task_options": {
          "max_probe_retries": 0,
          "fail_job_on_retries_exhausted": true
        }
      },
      "status": "completed",
      "status_details": {
        "progress": {
          "probes_total": 2,
          "probes_complete": 2,
          "probes_failed": 0
        },
        "runtime_probe_status": {
          "probe_name": "dan.AutoDANCached",
          "retry": 0,
          "max_probe_retries": 0
        }
      }
    }
  ],
  "pagination": {
    "page": 1,
    "page_size": 10,
    "current_page_size": 2,
    "total_pages": 1,
    "total_results": 2
  },
  "sort": "-created_at",
  "filter": {}
}

Pausing and Resuming a Job#

You can pause a job by sending a POST request to the /v1beta1/audit/jobs/{id}/pause endpoint.

Pausing a job stops the currently running probe and makes the results from completed probes available from the /v1beta1/audit/jobs/{id}/results endpoint. If no probes have completed, the list of result artifacts is empty.

client.beta.audit.jobs.pause(job_id=job_id)

curl -X POST "${AUDITOR_BASE_URL}/v1beta1/audit/jobs/${JOB_ID}/pause" \
    -H "Accept: application/json"

You can resume a job by sending a POST request to the /v1beta1/audit/jobs/{id}/resume endpoint.

client.beta.audit.jobs.resume(job_id=job_id)

curl -X POST "${AUDITOR_BASE_URL}/v1beta1/audit/jobs/${JOB_ID}/resume" \
    -H "Accept: application/json"

Cancel an Audit Job#

client.beta.audit.jobs.cancel(job_id=job_id)

curl -X POST "${AUDITOR_BASE_URL}/v1beta1/audit/jobs/${JOB_ID}/cancel" \
    -H "Accept: application/json"

Audit Job Schema#

The following table shows the schema for the audit job.

Field Name

Data Type

Description

name

string

Specifies the name of the job. Auditor does not require unique job names.

description

string

Specifies a description for the job.

namespace

string

Specifies the namespace for the job. The default value is default.

project

string

Specifies the project for the job.

spec.config (Required)

string

Specifies the audit configuration for the job.

You can specify <namespace>/<config-name> or the ID, such as audit_config-G7qVcW2F3z5t7wP9C4sxLi.

spec.target (Required)

string

Specifies the audit target for the job.

You can specify <namespace>/<target-name> or the ID, such as audit_target-2KbxQRDwjHZFNkZj5fzSca.

spec.task_options.max_probe_retries

integer

Specifies the maximum number of retries to perform for a failed probe.

The default value is 0 and specifies to never retry a failed probe.

Failed probes are not recorded in the report.html result file. If all probes for a job fail, the job reports a status of error even if fail_job_on_retries_exhausted is set to false.

spec.task_options.fail_job_on_retries_exhausted

boolean

When set to false, when a probe fails the maximum number of retries, Auditor moves on to the next probe.

The default value, true, specifies to stop the audit job and set the job status to failed.