Download this tutorial as a Jupyter notebook

Running a Simple Audit Job in Docker#

The following procedure demonstrates how to use Auditor to probe a model deployed via the Inference Gateway.

Prerequisites#

  • Quickstart installed and running

  • NGC API key for accessing NGC Catalog (if not already configured during quickstart setup). The API key does not require any special permissions.

  • At least 4GB of available RAM.

  • Sufficient disk space for generated artifacts (recommended: 10GB+).

Follow the quickstart instructions to install and start NeMo Platform, which includes NeMo Auditor and its dependencies.

With NeMo Platform running, follow the model provider instructions to set up a model provider that points at the model to audit. Create the provider in the default workspace and name it my-model-provider.

Procedure#

Perform the following steps.

Tip

Before proceeding, ensure NeMo Auditor is running. You can run curl http://localhost:8080/apis/audit/v2/info to check that the service is running.

  1. Set the base URL for the service in an environment variable:

    export NMP_BASE_URL=http://localhost:8080

  2. Create a configuration that runs only two probes and sends 32 requests in parallel. By running two probes, the job is less likely to encounter rate limiting.

    nmp audit configs create --workspace "default" \
  --name "demo-simple-config" \
  --system '{"parallel_attempts": 32, "lite": true}' \
  --run '{"generations": 7}' \
  --plugins '{"probe_spec": "dan.AutoDANCached,goodside.Tag"}' \
  --reporting '{"extended_detectors": false}'

  3. Create a target that accesses a model from through the inference gateway provider.

    nmp audit targets create --workspace "default" \
  --model "mistralai/mistral-7b-instruct-v0.3" \
  --name "demo-simple-target" \
  --type "nim" \
  --options '{"nim": { "skip_seq_start": "<think>", "skip_seq_end": "</think>", "max_tokens": 4000,
           "nmp_uri_spec": {"inference_gateway": { "workspace": "default", "provider": "my-model-provider" }}}}'

  4. Start the simple audit job with the target and config:

    nmp audit jobs create --workspace default \
  --name demo-simple-job \
  --spec '{ "config": "default/demo-simple-config", "target": "default/demo-simple-target" }'

  5. Get the audit job status.

    The job transitions from created to pending and then to active.

    nmp audit jobs get-status demo-simple-job

    Initially, the status shows 0 completed probes.

    If an unrecoverable error occurs, the status becomes error and the error_details field includes error messages from the service logs.

    Eventually, the status becomes completed.

  6. View the job logs. Viewing the logs can help you confirm the job is running correctly or assist with troubleshooting.

    nmp audit jobs get-logs demo-simple-job

    Logs show the probe attempts and transient errors. If the target model rate limits the probe attempts, the log includes the HTTP errors; however, the job status does not transition to error because the job can continue. If the job seems to run slowly but is still in the active state, the logs can help you understand if the job is slowed by rate limiting or other transient errors are causing the process to progress slowly.

  7. Download HTML report:

    nmp audit jobs results download --job demo-simple-job -o ./demo-simple-job-report.html report-html
On this page