garak#

This page contains all evaluation tasks for the garak harness.

Task	Description
garak	Task for running the default set of Garak probes. This variant uses the chat endpoint.
garak-completions	Task for running the default set of Garak probes. This variant uses the completions endpoint.

garak#

Task for running the default set of Garak probes. This variant uses the chat endpoint.

Container

Harness: garak

Container:

nvcr.io/nvidia/eval-factory/garak:26.01

Container Digest:

sha256:72514ac2c35f76fdb139b02f1c1d4159103969946a121592e50b129087dd455e

Container Arch: multiarch

Task Type: garak

Command

cat > garak_config.yaml << 'EOF'
{% if config.params.extra.seed is not none %}run:
  seed: {{config.params.extra.seed}}{% endif %}
plugins:
  {% if config.params.extra.probes is not none %}probe_spec: {{config.params.extra.probes}}{% endif %}
  extended_detectors: true
  target_type: {% if target.api_endpoint.type == "completions" %}nim.NVOpenAICompletion{% elif target.api_endpoint.type == "chat" %}nim.NVOpenAIChat{% endif %}
  target_name: {{target.api_endpoint.model_id}}
  generators:
    nim:
      uri: {{target.api_endpoint.url | replace('/chat/completions', '') | replace('/completions', '')}}
      {% if config.params.temperature is not none %}temperature: {{config.params.temperature}}{% endif %}
      {% if config.params.top_p is not none %}top_p: {{config.params.top_p}}{% endif %}
      {% if config.params.max_new_tokens is not none %}max_tokens: {{config.params.max_new_tokens}}{% endif %}
    skip_seq_start: {{config.params.extra.skip_seq_start}}
    skip_seq_end: {{config.params.extra.skip_seq_end}}
    
system:
  parallel_attempts: {{config.params.parallelism}}
  lite: false
EOF
{% if target.api_endpoint.api_key_name is not none %}
export NIM_API_KEY=${{target.api_endpoint.api_key_name}} &&
{% else %}
export NIM_API_KEY=dummy &&
{% endif %}
export XDG_DATA_HOME={{config.output_dir}} &&
garak --config garak_config.yaml --report_prefix=results

Defaults

framework_name: garak
pkg_name: garak
config:
  params:
    max_new_tokens: 150
    parallelism: 32
    temperature: 0.1
    top_p: 0.7
    extra:
      probes: null
      seed: 42
      skip_seq_start: <think>
      skip_seq_end: </think>
  supported_endpoint_types:
  - chat
  type: garak
target:
  api_endpoint:
    api_key_name: API_KEY

garak-completions#

Task for running the default set of Garak probes. This variant uses the completions endpoint.