> This page is for version 0.8.
> For other versions, use one of these documentation indexes:
> - Latest (default): https://docs.nvidia.com/nemo-helix/latest/llms.txt
> - 0.9: https://docs.nvidia.com/nemo-helix/v0.9/llms.txt
> - 0.8: https://docs.nvidia.com/nemo-helix/v0.8/llms.txt
> - 0.7: https://docs.nvidia.com/nemo-helix/v0.7/llms.txt
> - 0.6: https://docs.nvidia.com/nemo-helix/v0.6/llms.txt
> - 0.5: https://docs.nvidia.com/nemo-helix/v0.5/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo-helix/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo-helix/_mcp/server.

# Run an Audit Locally

This tutorial walks through running a single audit end-to-end with the NeMo garak plugin SDK. You will persist an audit configuration and a target, run the audit in-process, and inspect the resulting report artifacts.

**What you will learn:**

* Initialize NeMo Helix SDK and reach the `garak` resource.
* Create an `AuditConfig` selecting which garak probes to run.
* Create an `AuditTarget` pointing at a model through the Inference Gateway.
* Execute the audit locally with `garak.run(...)`.
* Read the JSONL, HTML, and hitlog report artifacts the run produces.

> **Tip**
>
> This tutorial takes approximately **10 minutes** to complete, plus however long garak takes to run the selected probes against your target.

## Prerequisites

* Install and start NeMo Helix using the [Setup guide](/nemo-helix/v0.8/documentation/get-started).
* Install garak in a Python virtual environment so the plugin can shell out to it. By default the plugin invokes `~/.garak/.venv/bin/python -m garak`. Override the interpreter path with `NEMO_GARAK_PYTHON` if your install lives elsewhere.
* Configure at least one Inference Gateway provider — this tutorial uses the default `nvidia-build` provider for NVIDIA Build (`https://integrate.api.nvidia.com/v1`), with a Build API key authorized to call the selected model. Any chat-completion-compatible provider works, but its model identifiers and credentials must match that provider. Step 3 lists registered providers and checks completion access before running garak. See [Inference Gateway](/nemo-helix/v0.8/documentation/vulnerability-scanning/targets/inference-gateway) for details on the `nhx_uri_spec` block used below.

---

## Key Concepts

* **`AuditConfig`** — Selects probes, detectors, and reporting settings for a garak run. Persisted in the entity store.
* **`AuditTarget`** — Identifies the model under test (generator class, model identifier, endpoint options). Persisted in the entity store.
* **In-process run** — `garak.run(...)` shells out to garak on the host running the SDK call, writes report artifacts under a temporary directory, and returns their file paths. There is no remote job submission.

---

## 1. Initialize the SDK

Connect to the running platform and reach the garak sub-resource:

```python
from nemo_garak.sdk import GarakPluginResource
from nemo_helix_plugin.client.client import NemoClient

client = NemoClient.from_config()
garak = GarakPluginResource(client)

print(garak.plugin_status())
# {'plugin': 'garak', 'status': 'ok', ...}
```

> **Note**
>
> These examples use the `default` workspace. If your current CLI context uses a different workspace, run `nemo config set --workspace default` first. To check, run `nemo config view`.

If `plugin_status()` raises a connection error, the platform is not running or your current CLI context points at the wrong deployment. Run `nemo config view` to check the context, and refer back to [Setup](/nemo-helix/v0.8/documentation/get-started) or [Connect with CLI Contexts](/nemo-helix/v0.8/documentation/reference/cli-reference/connect-to-deployments).

---

## 2. Create an Audit Configuration

Build a bounded configuration that runs one lightweight garak probe with one generation:

```python
from nemo_garak.entities import (
    AuditPluginsData,
    AuditReportData,
    AuditRunData,
    AuditSystemData,
)

config = garak.configs.create(
    workspace="default",
    name="quick-scan",
    description="Bounded scan using one probe and one generation.",
    system=AuditSystemData(lite=True, parallel_attempts=4),
    run=AuditRunData(generations=1),
    plugins=AuditPluginsData(probe_spec="goodside.Tag", detector_spec="auto"),
    reporting=AuditReportData(report_prefix="quick-scan"),
)
print(config.model_dump_json(indent=2))
```

For the full set of options on each sub-block, see [Configuration Schema](/nemo-helix/v0.8/documentation/vulnerability-scanning/configurations/schema). For probe selection syntax, see [Selecting Probes](/nemo-helix/v0.8/documentation/vulnerability-scanning/configurations/selecting-probes).

---

## 3. Create an Audit Target

First, list the providers registered in your deployment so you can pick one that exists:

```python
providers = client.models.list_providers(workspace="default")
for provider in providers.items():
    print(provider.name, provider.host_url)
```

Use a `provider` name from this output in the target below. This example pairs `nvidia-build` with the Build model ID `nvidia/nemotron-3.5-lightning-30b-a3b`. Do not add a second `nvidia/` prefix or reuse this ID with `nvidia-inference-api`: that provider has a different catalog and requires its own credentials.

Before creating the target, send one chat completion through the same provider route that garak will use. A READY provider or a model listed in its catalog does not guarantee that your API key has completion access:

```python
from nemo_helix_plugin.inference_gateway.client import InferenceGatewayClient
from nemo_helix_plugin.inference_gateway.types import JsonBody

completion = InferenceGatewayClient.from_client(client).provider_post(
    workspace="default",
    name="nvidia-build",
    trailing_uri="v1/chat/completions",
    body=JsonBody({
        "model": "nvidia/nemotron-3.5-lightning-30b-a3b",
        "messages": [{"role": "user", "content": "Reply with hello."}],
        "max_tokens": 1024,
    }),
).data()
content = completion["choices"][0]["message"].get("content")
if not content:
    raise RuntimeError("Completion returned no content; inspect the response before auditing.")
print(content)
```

If this request fails, resolve the provider/model or credential issue before starting the audit; see Troubleshooting below.

Define the model under test. This example uses garak's `nim.NVOpenAIChat` generator pointed at the `nvidia-build` provider:

```python
target = garak.targets.create(
    workspace="default",
    name="nemotron-3.5-lightning-30b",
    type="nim.NVOpenAIChat",
    model="nvidia/nemotron-3.5-lightning-30b-a3b",
    options={
        "nim": {
            "max_tokens": 1024,
            "nhx_uri_spec": {
                "inference_gateway": {
                    "workspace": "default",
                    "provider": "nvidia-build",
                },
            },
        },
    },
)
print(target.model_dump_json(indent=2))
```

The `nhx_uri_spec` sentinel inside `options.nim` tells the plugin to resolve a concrete URI from the Inference Gateway provider at run time. See [Inference Gateway](/nemo-helix/v0.8/documentation/vulnerability-scanning/targets/inference-gateway) for the full conflict rules and resolution behavior.

---

## 4. Run the Audit Locally

`run()` accepts either inline entities or name strings that reference entities in the entity store. Pass the names you persisted above:

```python
result = garak.run(
    config="quick-scan",
    target="nemotron-3.5-lightning-30b",
    workspace="default",
)

print(f"Audit status: {result['status']}")
if result["status"] == "failed":
    print(f"Audit failed: {result.get('error', 'unknown error')}")
else:
    print(result["probes_complete"], result["probes_failed"])
    # 1 0
    for name, ref in result["results"].items():
        print(f"{name}: {ref['artifact_url']}")
    # report-jsonl: file:///var/folders/.../results/report-jsonl
    # report-html: file:///var/folders/.../results/report-html
    # report-hitlog-jsonl: file:///var/folders/.../results/report-hitlog-jsonl
```

If you prefer to skip the entity-store roundtrip, pass the inline `AuditConfig` and `AuditTarget` objects from the previous two steps directly:

```python
result = garak.run(config=config, target=target, workspace="default")
```

---

## 5. Read the Results

`run()` returns `file://` URLs for whichever of the three report types garak produced. Parse them with `urllib.parse.urlparse` to recover local paths, then load them as needed.

Load the JSONL probe-by-probe summary:

```python
import json
from pathlib import Path
from urllib.parse import urlparse

def url_to_path(url: str) -> Path:
    return Path(urlparse(url).path)

jsonl_path = url_to_path(result["results"]["report-jsonl"]["artifact_url"])
records = [json.loads(line) for line in jsonl_path.read_text().splitlines() if line.strip()]
print(f"Loaded {len(records)} report entries.")
```

Summarize hits per probe from the hitlog:

```python
from collections import Counter

hitlog_ref = result["results"].get("report-hitlog-jsonl")
if hitlog_ref is None:
    print("No hits recorded — every probe passed.")
else:
    hitlog_path = url_to_path(hitlog_ref["artifact_url"])
    hits = [json.loads(line) for line in hitlog_path.read_text().splitlines() if line.strip()]
    by_probe = Counter(entry.get("probe") for entry in hits)
    for probe, count in by_probe.most_common():
        print(f"{probe}: {count} hits")
```

The HTML report at `result["results"]["report-html"]["artifact_url"]` is the most human-friendly summary — open it in a browser to see grouped pass/fail counts and per-probe details.

---

## 6. Clean Up

Delete the entities you created:

```python
garak.configs.delete(workspace="default", name="quick-scan")
garak.targets.delete(workspace="default", name="nemotron-3.5-lightning-30b")
```

Report artifacts are left in place under the scheduler's temporary directory; the OS reaps them eventually.

---

## Troubleshooting

**`FileNotFoundError: garak interpreter not found at ...`**
: The plugin couldn't find a garak install. Either install garak at `~/.garak/.venv/bin/python`, or set `NEMO_GARAK_PYTHON` to the absolute path of a Python interpreter that has garak installed.

**`RuntimeError: Failed to resolve inference gateway provider '<workspace>/<provider>'`**
: The `nhx_uri_spec` block in your target's options references an Inference Gateway provider that doesn't exist. List your providers with `client.models.list_providers(workspace="default")` and update the target to reference an existing one.

**HTTP 403, 404, or 424 during the completion preflight**
: Check the provider, its API base URL, the exact model ID in its served-model catalog, and the provider's configured API key. An upstream 403 (`Authorization failed` or `key_model_access_denied`) indicates an authorization/access failure; confirm both the model ID and the key's access with that provider. An upstream 404 can indicate an incorrect model ID, such as a duplicated `nvidia/` prefix on NVIDIA Build. The platform may wrap an upstream failure as HTTP 424; inspect the response body for the upstream status and error. Do not retry the audit until the completion preflight succeeds.

**HTTP 504 during the completion preflight**
: The upstream completion timed out. Provider readiness and model discovery are not successful completion tests. Retry the preflight or choose another model available to your provider and credential before running garak.

**`status == "failed"` with empty `results`**
: The audit could not complete any probe. Inspect `result.get("error")` for the failure reason, and verify the model is reachable through the provider you configured.

**Where do the report files live?**
: The scheduler writes garak output under `<temp-dir>/garak/<reporting.report_dir>/<reporting.report_prefix>.*`, then copies the produced files into the local results directory referenced by each `artifact_url`. For a local run this is a temporary directory under the system's `$TMPDIR`.

---

## Summary and Next Steps

You created an `AuditConfig` and `AuditTarget`, ran a single audit locally with `garak.run(...)`, and loaded the resulting reports.

* For the full SDK surface — including async variants — see [SDK Resources](/nemo-helix/v0.8/documentation/vulnerability-scanning/sdk-resources).
* For more probe selection options, see [Selecting Probes](/nemo-helix/v0.8/documentation/vulnerability-scanning/configurations/selecting-probes).
* For other generator types (OpenAI-compatible endpoints, REST targets, test targets), see [Audit Targets](/nemo-helix/v0.8/documentation/vulnerability-scanning/targets).