Run and Manage Audit Jobs#

After you create an audit target and an audit configuration, you are ready to run an audit job.

Prerequisites#

  1. Create a new target or find an existing target for the audit and record the ID.

  2. Create a new configuration or find an existing configuration for the audit and record the ID. Alternatively, you can specify default/default for the config argument to use the default configuration.

Create an Audit Job#

Tip

Before proceeding, ensure NeMo Auditor is running and accessible. If you are running NeMo Auditor in a Kubernetes cluster, you can use port-forwarding to access the service locally: kubectl port-forward svc/auditor 5000:5000 -n <namespace>

  1. Set AUDITOR_BASE_URL to specify the service:

    $ export AUDITOR_BASE_URL=http://localhost:5000

  2. Create the job with the basic target and basic config:

    import os
from nemo_microservices import NeMoMicroservices

client = NeMoMicroservices(base_url=os.getenv("AUDITOR_BASE_URL"))

job = client.beta.audit.jobs.create(
    name="demo-basic-job",
    project="demo",
    spec={
        "config": "default/default",
        "target": "default/demo-local-llm-target"
    },
)
print(job.id)
print(job.model_dump_json(indent=2))

    curl -X POST "${AUDITOR_BASE_URL}/v1beta1/audit/jobs" \
  -H "Accept: application/json" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "demo-basic-job",
    "project": "demo",
    "spec": {
        "config": "default/default",
        "target": "default/demo-local-llm-target"
    }
}' | jq

    Example Output

    job-pcanyxxhidh55rvhdwnh9y

{
  "name": "demo-basic-job",
  "spec": {
    "config": "default/demo-basic-config",
    "target": "default/demo-local-llm-target"
  },
  "id": "job-pcanyxxhidh55rvhdwnh9y",
  "created_at": "2025-09-24T14:22:01.853151",
  "custom_fields": null,
  "description": null,
  "error_details": null,
  "namespace": "default",
  "ownership": null,
  "project": "demo",
  "status": "created",
  "status_details": {},
  "updated_at": "2025-09-24T14:22:01.853157"
}

    {
  "id": "job-4imupzhbxjznnjkhv7u7aq",
  "name": "demo-basic-job",
  "description": null,
  "project": "demo",
  "namespace": "default",
  "created_at": "2025-09-24T14:28:16.976048",
  "updated_at": "2025-09-24T14:28:16.976053",
  "spec": {
    "config": "default/demo-basic-config",
    "target": "default/demo-local-llm-target"
  },
  "status": "created",
  "status_details": {},
  "error_details": null,
  "ownership": null,
  "custom_fields": null
}

After you create the job, check the status to ensure it becomes active.

Get Audit Job Status#

A Job can report the following statuses:

created

The job exists but has not yet been submitted to a job executor. This is the initial state for all new jobs.

pending

The job is waiting for active execution.

active

The job is currently executing its tasks.

error

The job terminated due to an unrecoverable error and cannot be retried.

cancelling

The Jobs microservice is attempting to terminate the job gracefully.

cancelled

The job is terminated as a result of a cancellation request.

pausing

The Jobs microservice is attempting to pause the job.

paused

The job is paused as a result of a pause request.

resuming:

The paused job is resuming to the active state as a result of a resume request.

completed

The job finished all its steps successfully without errors.

Send a GET request to the /v1beta1/audit/jobs/{id}/status endpoint.

import os
from nemo_microservices import NeMoMicroservices

client = NeMoMicroservices(base_url=os.getenv("AUDITOR_BASE_URL"))

status = client.beta.audit.jobs.get_status(job_id)
print(status.model_dump_json(indent=2))

  curl "${AUDITOR_BASE_URL}/v1beta1/audit/jobs/${JOB_ID}/status" \
    -H "Accept: application/json" | jq

Example Output

{
  "error_details": null,
  "job_id": "job-pcanyxxhidh55rvhdwnh9y",
  "status": "completed",
  "status_details": {
    "progress": {
      "probes_total": 2,
      "probescomplete": 2
    }
  },
  "steps": [
    {
      "error_details": {},
      "name": "audit",
      "status": "completed",
      "status_details": {},
      "tasks": [
        {
          "id": "509907e522b34104b0b31a424b14202a",
          "error_details": {},
          "error_stack": null,
          "status": "completed",
          "status_details": {}
        }
      ]
    }
  ]
}

{
  "job_id": "job-4imupzhbxjznnjkhv7u7aq",
  "status": "completed",
  "status_details": {
    "progress": {
      "probes_total": 2,
      "probescomplete": 2
    }
  },
  "error_details": null,
  "steps": [
    {
      "name": "audit",
      "status": "completed",
      "status_details": {},
      "error_details": {},
      "tasks": [
        {
          "id": "3ef607c222b5489089b18ad4b5bb7d11",
          "status": "completed",
          "status_details": {},
          "error_details": {},
          "error_stack": null
        }
      ]
    }
  ]
}

After you confirm a job is active, view the audit job log to check for progress.

Get an Audit Job Log#

You can retrieve a log of the job progress. The log is produced by garak for each probe.

  • Send a GET request to the /v1beta1/audit/jobs/{id}/logs endpoint.

    client = NeMoMicroservices(base_url=os.getenv("AUDITOR_BASE_URL"))
logs = client.beta.audit.jobs.get_logs(job_id)
print("".join(log.message for log in logs.data[-10:]))

        curl "${AUDITOR_BASE_URL}/v1beta1/audit/jobs/${JOB_ID}/logs" \
      -H "Accept: text/plain"
    Partial Job Log
    2025-09-24 14:39:01,356  DEBUG  connect_tcp.started host='local-llm' port=8000 local_address=None timeout=5.0 socket_options=None
2025-09-24 14:39:01,356  DEBUG  send_request_headers.complete
2025-09-24 14:39:01,356  DEBUG  Sending HTTP Request: POST http://local-llm:8000/v1/chat/completions
2025-09-24 14:39:01,356  DEBUG  send_request_body.started request=<Request [b'POST']>
2025-09-24 14:39:01,356  DEBUG  connect_tcp.complete return_value=<httpcore._backends.sync.SyncStream object at 0x7f33d51c1f50>
2025-09-24 14:39:01,356  DEBUG  send_request_body.complete
2025-09-24 14:39:01,356  DEBUG  receive_response_headers.started request=<Request [b'POST']>
2025-09-24 14:39:01,356  DEBUG  send_request_headers.started request=<Request [b'POST']>
2025-09-24 14:39:01,356  DEBUG  connect_tcp.complete return_value=<httpcore._backends.sync.SyncStream object at 0x7f33d51c1c10>
2025-09-24 14:39:01,356  DEBUG  connect_tcp.started host='local-llm' port=8000 local_address=None timeout=5.0 socket_options=None

List all Jobs#

The /v1beta1/audit/jobs endpoint lists all jobs since the microservice was last started.

jobs = client.beta.audit.jobs.list()
print("\n".join([job.model_dump_json(indent=2) for job in jobs]))

curl -X GET "${AUDITOR_BASE_URL}/v1beta1/audit/jobs" \
  -H "Accept: application/json" | jq

Example Output

{
  "name": "demo-basic-job",
  "spec": {
    "config": "default/demo-basic-config",
    "target": "default/demo-local-llm-target"
  },
  "id": "job-nr2419u199xqfinjrzcha2",
  "created_at": "2025-09-24T14:38:50.781540",
  "custom_fields": null,
  "description": null,
  "error_details": null,
  "namespace": "default",
  "ownership": null,
  "project": "demo",
  "status": "completed",
  "status_details": {
    "progress": {
      "probes_total": 2,
      "probescomplete": 2
    }
  },
  "updated_at": "2025-09-24T14:38:50.781544"
}

[
  {
    "id": "job-4imupzhbxjznnjkhv7u7aq",
    "name": "demo-basic-job",
    "description": null,
    "project": "demo",
    "namespace": "default",
    "created_at": "2025-09-24T14:28:16.976048",
    "updated_at": "2025-09-24T14:28:16.976053",
    "spec": {
      "config": "default/demo-basic-config",
      "target": "default/demo-local-llm-target"
    },
    "status": "completed",
    "status_details": {
      "progress": {
        "probes_total": 2,
        "probescomplete": 2
      }
    },
    "error_details": null,
    "ownership": null,
    "custom_fields": null
  },
  {
    "id": "job-pcanyxxhidh55rvhdwnh9y",
    "name": "demo-basic-job",
    "description": null,
    "project": "demo",
    "namespace": "default",
    "created_at": "2025-09-24T14:22:01.853151",
    "updated_at": "2025-09-24T14:22:01.853157",
    "spec": {
      "config": "default/demo-basic-config",
      "target": "default/demo-local-llm-target"
    },
    "status": "completed",
    "status_details": {
      "progress": {
        "probes_total": 2,
        "probescomplete": 2
      }
    },
    "error_details": null,
    "ownership": null,
    "custom_fields": null
  }
]

Pausing and Resuming a Job#

You can pause a job by sending a POST request to the /v1beta1/audit/jobs/{id}/pause endpoint.

Pausing a job stops the currently running probe and makes the results from completed probes available from the /v1beta1/audit/jobs/{id}/results endpoint. If no probes have completed, the list of result artifacts is empty.

client.beta.audit.jobs.pause(job_id=job_id)

curl -X POST "${AUDITOR_BASE_URL}/v1beta1/audit/jobs/${JOB_ID}/pause" \
    -H "Accept: application/json"

You can resume a job by sending a POST request to the /v1beta1/audit/jobs/{id}/resume endpoint.

client.beta.audit.jobs.resume(job_id=job_id)

curl -X POST "${AUDITOR_BASE_URL}/v1beta1/audit/jobs/${JOB_ID}/resume" \
    -H "Accept: application/json"

Cancel an Audit Job#

client.beta.audit.jobs.cancel(job_id=job_id)

curl -X POST "${AUDITOR_BASE_URL}/v1beta1/audit/jobs/${JOB_ID}/cancel" \
    -H "Accept: application/json"