Error Threshold File#

When you run a health check (dcgmi health) or a diagnostic (dcgmi diag), DCGM compares readings from the GPU against these thresholds and reports an error when a reading breaches one. Those thresholds were previously fixed inside the executables; DCGM now publishes them in a file, error_thresholds.yaml, so that you can read the value behind every such check and change a threshold without rebuilding DCGM or waiting for a new release. Every part of DCGM that applies one of these thresholds reads the same file, so a change applies wherever that threshold is used.

The installed file supplies NVIDIA’s defaults. Edit a threshold to suit a fleet whose tolerance differs, for example to report a GPU sooner than the default would. The file cannot add a new threshold check, change which fields a check reads, or change how a value is compared. Each entry carries a comment describing what it covers and which part of DCGM checks it.

YAML file structure#

A complete file has the following shape:

version: "<DCGM-release>"
spec: dcgm-error-thresholds-v1
common_thresholds:
  <threshold-name>:                    # an entry with a single-field comparison
    <GT | GTE | EQ | NOT_EQ>:
      value: <number>
      field_id: <DCGM_FI_*>
    metadata:
      system: <system>
      severity: <severity>
  <threshold-name>:                    # an entry with a multi-field comparison
    <SUM_GTE | ANY_GT | ANY_GTE>:
      value: <number>
      field_ids: [<DCGM_FI_*>, <DCGM_FI_*>, ...]
    metadata:
      system: <system>
      severity: <severity>

Each entry holds exactly one comparison. A single-field comparison reads one field from field_id; a multi-field comparison reads two or more fields from field_ids. See the comparison values below.

Names, comparisons, field identifiers, and the system and severity values are case-sensitive. Write them exactly as shown on this page.

Top-level fields#

Error threshold file top-level fields#
Field	Type	Required	Meaning
`version`	String	Yes	DCGM release that supplied the file. Recorded for debugging.
`spec`	String	Yes	Format identifier. The current value is `dcgm-error-thresholds-v1`. DCGM refuses to start on a file that declares any other spec.
`common_thresholds`	Mapping	Yes	The threshold entries, each keyed by its threshold name. DCGM refuses to start if an entry a check needs is absent.

Threshold entry fields#

Threshold entry fields#
Field	Type	Tunable	Meaning
Threshold name (the key)	String	No	Identifies the entry to the threshold check that reads it. Renaming an entry stops that check from finding it, so DCGM refuses to start.
Comparison (the key, for example `GT`)	Mapping	No	How a reading is compared against `value`. It holds the fields below. See the comparison values below.
`metadata`	Mapping	No	Describes the threshold check, so that a reader can tell what a value controls before changing it.

Comparison fields#

Comparison fields#
Field	Type	Tunable	Meaning
`value`	Number	Yes	The number a reading is compared against. This is the only field you are expected to change.
`field_id`	String	No	The field a single-field comparison reads. See Field Identifiers.
`field_ids`	Sequence	No	The fields a multi-field comparison reads: two or more, each listed once. See Field Identifiers.

Metadata fields#

Metadata fields#
Field	Type	Meaning
`system`	String	The hardware area a breach belongs to: `memory`, `nvlink`, `nvswitch`, `thermal`, `power`, or `other`. See the error categories in Diagnostic Errors.
`severity`	String	How serious a breach is: `monitor`, `isolate`, `triage`, `config`, `reset`, or `none`. See the error severities in Diagnostic Errors.

Comparison values#

Comparison values#
Value	Reads	DCGM reports an error when
`GT`	`field_id`	The field value is greater than `value`.
`GTE`	`field_id`	The field value is greater than or equal to `value`.
`EQ`	`field_id`	The field value equals `value`.
`NOT_EQ`	`field_id`	The field value differs from `value`.
`SUM_GTE`	`field_ids`	The total of the listed fields is greater than or equal to `value`.
`ANY_GT`	`field_ids`	Any one of the listed fields is greater than `value`.
`ANY_GTE`	`field_ids`	Any one of the listed fields is greater than or equal to `value`.

The comparison is fixed for each check and is described here so that you can see what a value means. Changing it in the file does not change how DCGM compares a reading.

What DCGM compares value against depends on the entry. It can be a single sampled value, a rate in errors per second, or the change in a counter over a fixed window of one minute. The comment on each entry says what that entry compares. You configure value. You do not configure the comparison, the rate basis, or the window length.

Validation#

DCGM refuses to start when it cannot use the file it found, and the failure names the file and what was wrong with it. A file is rejected when it:

is missing, empty, a symlink, or larger than 10 MB;
cannot be parsed as YAML, or holds something other than key-value pairs at the top level;
declares a spec this DCGM does not read, or omits version;
holds common_thresholds as anything other than a mapping of threshold names to entries;
repeats a threshold name;
has an entry without metadata, or without exactly one comparison beside it;
gives a single-field comparison anything other than one field_id, or a multi-field comparison fewer than two field_ids or the same field twice;
names a comparison, field identifier, system, or severity value that DCGM does not recognize, including one written in a different case; or
gives a value that is not a finite number.

DCGM also refuses to start when an entry a threshold check needs is absent. An entry you add yourself is loaded, but no check reads it.

Schema#

DCGM installs a schema that describes this file in the same directory as the installed defaults, error_thresholds.schema.yaml (/usr/share/datacenter-gpu-manager-4/ for a standard install). It is a JSON Schema (draft 2020-12) written in YAML, and it changes together with the spec the file declares.

Validate an edited copy against the schema before you restart the host engine, so that a mistake is reported while you edit rather than as a failure to start. Any JSON Schema validator that reads YAML works, for example the Python jsonschema and PyYAML packages:

$ python3 -c 'import sys, yaml, jsonschema; jsonschema.validate(yaml.safe_load(open(sys.argv[2])), yaml.safe_load(open(sys.argv[1])))' \
    /usr/share/datacenter-gpu-manager-4/error_thresholds.schema.yaml \
    /etc/datacenter-gpu-manager-4/error_thresholds.yaml

Adjust the first path if DCGM is installed outside /usr, and the last path if your copy is not under /etc. The command prints nothing when the copy is valid. Otherwise it fails with an error that names the first problem it found and where in the file it is.

The schema requires every threshold DCGM checks, with the comparison and fields the installed file gives it, and rejects keys that the format does not define. It cannot detect a repeated threshold name or a value of .nan; DCGM still rejects both when it reads the file.

Examples#

A default entry. DCGM reports an error when a GPU’s volatile double-bit ECC error count is greater than zero:

volatile_dbe:
  GT:
    value: 0
    field_id: DCGM_FI_DEV_ECC_DBE_VOL_TOTAL
  metadata:
    system: memory
    severity: isolate

An entry whose reading is a total. DCGM reports an error when the retired pages from double-bit and single-bit errors together reach the value:

max_retired_pages:
  SUM_GTE:
    value: 63
    field_ids: [DCGM_FI_DEV_PAGE_RETIRED_DBE_TOTAL, DCGM_FI_DEV_PAGE_RETIRED_SBE_TOTAL]
  metadata:
    system: memory
    severity: isolate

An entry checked field by field. DCGM reports an error when any one of the NVLink link recovery counters is greater than zero:

nvlink_blackwell_error:
  ANY_GT:
    value: 0
    field_ids: [DCGM_FI_DEV_NVLINK_RECOVERY_SUCCESSFUL_TOTAL, DCGM_FI_DEV_NVLINK_RECOVERY_FAILED_TOTAL]
  metadata:
    system: nvlink
    severity: isolate

To report a GPU only after it accumulates more retired pages than the default allows, change value in your copy and leave the rest of the entry alone:

max_retired_pages:
  SUM_GTE:
    value: 128
    field_ids: [DCGM_FI_DEV_PAGE_RETIRED_DBE_TOTAL, DCGM_FI_DEV_PAGE_RETIRED_SBE_TOTAL]
  metadata:
    system: memory
    severity: isolate

An edited value takes effect once DCGM reads the file that holds it, as described below.

File locations and precedence#

At start, DCGM searches these locations in order and reads the first file it finds:

Error threshold file resolution#
Search order	Source	Description
1	`DCGM_ERROR_THRESHOLDS_FILE` environment variable	The path it names, which overrides every other source.
2	`/etc/datacenter-gpu-manager-4/error_thresholds.yaml`	Your edited copy of the installed file. Create it yourself; packages never write to it, so an upgrade cannot overwrite your values.
3	`share/datacenter-gpu-manager-4/error_thresholds.yaml` relative to the installed package root	The installed defaults, when DCGM is installed outside `/usr`.
4	`/usr/share/datacenter-gpu-manager-4/error_thresholds.yaml`	The installed defaults.

At start the host engine logs the file it read, that file’s version, and every threshold it loaded, so the log shows which file won and the values DCGM is enforcing. These are INFO messages and the default log level is ERROR, so start the host engine with --log-level INFO to see them.

Changing a threshold#

DCGM reads the file only at start and holds those values for the life of the host engine process. There is no hot reload, so every change takes effect on the next restart.

To change a threshold, copy the installed file, edit the copy, and restart the host engine so it reads the new values. Name the copy in DCGM_ERROR_THRESHOLDS_FILE to keep it at a path of your choice, or place it at /etc/datacenter-gpu-manager-4/ for DCGM to find it without the variable.

The diagnostic loads the file as well, and checks that it read the same file the host engine did, so that the two never report against different thresholds. When they differ, the diagnostic stops and reports the mismatch. This happens when the file changed after the host engine read it, so restart the host engine and run the diagnostic again.

Using the `DCGM_ERROR_THRESHOLDS_FILE` environment variable#

$ sudo mkdir -p /opt/dcgm
$ sudo cp /usr/share/datacenter-gpu-manager-4/error_thresholds.yaml /opt/dcgm/
$ sudo $EDITOR /opt/dcgm/error_thresholds.yaml   # change a threshold with the editor of your choice
$ sudo systemctl edit nvidia-dcgm.service

In the editor that opens, add:

[Service]
Environment="DCGM_ERROR_THRESHOLDS_FILE=/opt/dcgm/error_thresholds.yaml"

Saving the drop-in reloads the systemd configuration, but the running host engine keeps its old values until it restarts:

$ sudo systemctl restart nvidia-dcgm

Using the `/etc` directory#

$ sudo mkdir -p /etc/datacenter-gpu-manager-4
$ sudo cp /usr/share/datacenter-gpu-manager-4/error_thresholds.yaml \
    /etc/datacenter-gpu-manager-4/
$ sudo $EDITOR /etc/datacenter-gpu-manager-4/error_thresholds.yaml   # change a threshold
$ sudo systemctl restart nvidia-dcgm

Upgrades#

An upgrade replaces the installed defaults. It does not touch a copy you created under /etc, and that copy continues to take precedence.

A release can add a threshold for a new check. If your copy predates that release, it lacks the new entry and DCGM refuses to start, naming the file and the missing entry. A release can also change the format. Your copy then declares a spec this DCGM does not read, and DCGM refuses to start, naming both values. Either way, copy the new installed file again and re-apply your edits. Comparing your copy against the installed one after an upgrade shows what changed:

$ diff /usr/share/datacenter-gpu-manager-4/error_thresholds.yaml \
    /etc/datacenter-gpu-manager-4/error_thresholds.yaml