About Auditing Models#

Important

NVIDIA NeMo Auditor is released with early access availability and is subject to limited support and potential API changes in future releases.

NVIDIA NeMo Auditor audits LLMs by running audit jobs that probe the model with a variety of prompts to identify vulnerabilities. You can use the results to assist with assessing model and system safety.

Typical Audit Workflow#

A typical audit workflow looks like the following:

  1. Create an audit target for a base model.

  2. Create an audit configuration.

  3. Run an audit job.

    1. Check the job status to ensure it becomes active.

    2. Periodically get the job log and interpret the log to determine progress.

  4. View the audit results.

  5. After fine-tuning the model, create an audit target for the fine-tuned model.

  6. Run another audit job, using the new target and the original audit configuration.

  7. View the results and compare.

Task Guides#

The following guides provide detailed information on how to perform common tasks.

Targets

Create targets for audit jobs.

Managing Audit Configurations
Configurations

Create configurations for audit jobs.

Managing Audit Configurations
Jobs

Create and run audit jobs to assess model and system safety.

Run and Manage Audit Jobs
Results

Get the results of your audit jobs.

Viewing Audit Job Results

References#

Target Schema

Reference documentation for the JSON schema used to define audit targets

Schema for Audit Targets
Config Schema

Reference documentation for the JSON schema used to define audit configurations

Schema for Audit Configurations