dcgmi policy#

NAME#

dcgmi-policy - configure and observe DCGM field-based and legacy policies

SYNOPSIS#

dcgmi policy --create --field <field-id> --name <name>
             --operator <operator> [--threshold <value>]
             [--entities <gpu-list|all>] [--channels <channels>]
             [--rate-limit <seconds>] [--disabled] [--json]
dcgmi policy --list [--json]
dcgmi policy --get --field <field-id> [--policy-id <id>] [--json]
dcgmi policy --modify --field <field-id> --policy-id <id> <change>...
dcgmi policy --delete (--field <field-id> --policy-id <id>|--all)
dcgmi policy --enable|--disable --field <field-id> --policy-id <id>
dcgmi policy --import <path>
dcgmi policy --watch [--field <field-id>] [--since <usec>]
             [--max-violations <count>] [--interval <seconds>]
             [--once] [--json]
dcgmi policy --get [--group <group-id>] [--verbose] [--json]
             [--host <address>]
dcgmi policy --set <action,validation> [--group <group-id>]
             <condition>... [--host <address>]
dcgmi policy --clear [--group <group-id>] [--host <address>]
dcgmi policy --reg [--group <group-id>] [--host <address>]

All operations accept --host <address>. Deprecated legacy operations are documented in DEPRECATED LEGACY POLICY.

DESCRIPTION#

Field-based policies compare a supported numeric GPU field with a configured condition and record matching violations. Policies can target all GPUs or selected GPU entity IDs, can be enabled or disabled, and are persisted by the host engine. Use --watch to read violations or the field-policy API for programmatic notifications.

FIELD POLICY OPERATIONS#

--create: Create a policy. --field, --name, and --operator are required. --threshold is also required unless the operator is changed.
--list: List all field-based policies.
--get --field <field-id> [--policy-id <id>]: Display policies for one field, or one policy when its ID is also given.
--modify --field <field-id> --policy-id <id> <change>...: Change one or more of --name, --threshold, --operator, --entities, --channels, or --rate-limit. The monitored field cannot be changed.
--delete --field <field-id> --policy-id <id>: Delete one policy. Use --delete --all to delete every field policy.
--enable and --disable: Change whether the policy evaluates samples and records violations. Both operations require --field and --policy-id.
--import <path>: Replace the host engine’s field policies from a YAML file located on the host-engine system. See YAML import format for path restrictions, replacement semantics, and the import format.
--watch: Poll and display violations. The command continues until interrupted unless --once is specified.

FIELD POLICY CONFIGURATION#

-f <field-id>, --field <field-id>: Select a supported numeric DCGM field ID.
--policy-id <id>: Select the unsigned policy ID returned when the policy was created.
--name <name>: Set the policy’s display name.
--operator <operator>: Select >, >=, <, <=, ==, !=, or changed. Text aliases such as gt, ge, lt, le, eq, ne, and the corresponding uppercase YAML names are also accepted.
--threshold <value>: Set the numeric comparison threshold. It is not used by changed.
--entities <gpu-list|all>: Select comma-separated GPU entity IDs or every GPU. Omitting this option selects every GPU.
--channels <channels>: Select a comma-separated combination of console, file, and callback. The default is console.
--rate-limit <seconds>: Set the minimum interval between notifications for one entity. The default is 60 seconds; zero disables rate limiting.
--disabled: Create the policy in the disabled state.

WATCHING AND OUTPUT#

--since <usec>: Return violations at or after this nonnegative timestamp, expressed in microseconds since 1970. The default is zero.
--max-violations <count>: Limit each retrieval. The value must be greater than zero.
--interval <seconds>: Set the polling interval for a continuous watch. The default is one second and the value must be greater than zero.
--once: Retrieve one batch and exit.
-j, --json: Format supported create, list, get, and watch output as JSON. A continuous JSON watch writes one JSON document per line; --once writes an array.
--host <address>: Connect to the specified host engine.

YAML import format#

The path passed to --import is interpreted by the host engine selected with --host; it is not opened by the dcgmi process. Use an absolute path that exists on that host. By default, the resolved path must be below /etc/datacenter-gpu-manager-4/ or /var/log/nvidia-dcgm/. The final path cannot be a symbolic link, and the file cannot exceed 10 MiB.

A successful import replaces the complete field-policy registry; it does not merge entries with existing policies. DCGM then saves the replacement to the host engine’s configured policy persistence file.

An import document uses schema version 1 and a policies sequence. Each policy requires fieldId, name, operator, threshold, channels, and status. The threshold is required even when the operator is CHANGED. rateLimitSeconds is optional and defaults to 60.

entities is an optional sequence of mappings. Each mapping must identify a GPU with entityGroupId: 1 and an unsigned entityId. Omitting entities or using an empty sequence selects all GPUs.

schema_version: 1
policies:
  - fieldId: 150
    name: GPU temperature warning
    operator: GREATER_THAN
    threshold: 85.0
    entities:
      - {entityGroupId: 1, entityId: 0}
    channels: [CONSOLE]
    status: ENABLED
    rateLimitSeconds: 60

Operator names are EQUAL, NOT_EQUAL, GREATER_THAN, LESS_THAN, GREATER_THAN_OR_EQUAL, LESS_THAN_OR_EQUAL, and CHANGED. Channel names are CONSOLE, FILE, and CALLBACK; at least one channel is required. Use the exact status ENABLED or DISABLED. Only ENABLED enables a policy; other strings are currently accepted as the disabled state.

Only one policy for a field ID can be loaded. A duplicate field ID causes batch validation to fail. If a nonempty sequence contains no valid entries, import fails and the existing registry is retained.

EXAMPLES#

Create a temperature policy for every GPU:

$ dcgmi policy --create --field 150 --name "GPU temperature warning" \
    --operator '>' --threshold 85 --entities all

List policies and inspect the policy ID returned by creation:

$ dcgmi policy --list
$ dcgmi policy --get --field 150 --policy-id <policy-id>

Read one batch of temperature-policy violations as JSON:

$ dcgmi policy --watch --field 150 --once --json

DEPRECATED LEGACY POLICY#

Legacy description#

The legacy interface manages a GPU group’s violation policy. A policy identifies conditions that can produce notifications. It also contains action and validation fields that DCGM stores and reports but does not execute.

The command can remain attached to the host engine and print violation notifications as they arrive. The default group is the built-in all-GPU group.

Legacy options#

Operation selection#

--get

Display the current policy for the selected group.

--set <action,validation>

Replace the current policy. The two comma-separated digits populate the action and validation fields:

Action 0 represents no action. Action 1 stores the deprecated GPU reset selection.
Validation 0 represents no validation. 1, 2, and 3 store the short, medium, and long system-validation selections, respectively.

At least one condition option is required. Nonzero action and validation values have no operational effect in the policy module.

--clear

Clear all conditions and stored response fields from the selected group’s policy.

--reg

Register for every supported policy condition and wait indefinitely for notifications. End the listener with an interrupt or termination signal.

Conditions#

-M <count>, --maxpages <count>: Trigger after the number of retired pages reaches the nonnegative integer count.
-T <degrees-c>, --maxtemp <degrees-c>: Trigger when a GPU reaches the specified nonnegative integer temperature in degrees Celsius.
-P <watts>, --maxpower <watts>: Trigger when a GPU reaches the specified nonnegative integer power level in watts.
-e, --eccerrors: Trigger on double-bit ECC errors.
-p, --pcierrors: Trigger on PCIe errors or replays.
-n, --nvlinkerrors: Trigger on NVLink errors.
-x, --xiderrors: Trigger on XID errors.

Targets and output#

-g <group-id>, --group <group-id>: Select a persistent GPU group. The default is the built-in all-GPU group.
-v, --verbose: With --get, display the policy separately for every GPU in the group.
-j, --json: With --get, format the policy body as JSON. DCGM also writes a plain-text heading before the document and can append a homogeneity notice. See JSON output.
--host <address>: Connect to the specified host engine.
-h, --help: Print generated usage information and exit.

Legacy policy condition reference#

Conditions supplied to one --set are combined: an observed violation of any selected condition is eligible for notification.

Legacy policy conditions#
Option	Threshold	Violation reported
`--maxpages <count>`	Caller-supplied, nonnegative integer retired-page count	The retired-page count reaches the limit; the notification reports the single-bit and double-bit retirement counts
`--maxtemp <degrees-c>`	Caller-supplied, nonnegative integer temperature in degrees Celsius	Measured GPU temperature reaches the limit
`--maxpower <watts>`	Caller-supplied, nonnegative integer power in watts	Measured GPU power reaches the limit
`--eccerrors`	Internal fixed criterion	A double-bit ECC error
`--pcierrors`	Internal fixed criterion	PCIe errors or replay events
`--nvlinkerrors`	Internal fixed criterion	An NVLink error counter violates DCGM’s criterion
`--xiderrors`	The driver event itself	An XID error is detected

The policy API’s internal criteria are not configurable through dcgmi. See Policy for condition categories and Policies for the programmatic interface.

Legacy state and side effects#

--set replaces the selected group’s complete policy; it does not add to an existing policy. The policy remains after this client disconnects, but is state in the running host engine rather than durable configuration. Use --get --verbose to detect GPUs in the group whose policy differs from the summary.

Action and validation are stored fields. --get displays their configured values, but DCGM does not dispatch a GPU reset, run a diagnostic, or send an action-completion notification after a policy violation. Use a notification consumer to coordinate recovery and invoke diagnostics explicitly; see dcgmi diag for the diagnostic interface.

--set does not start the policy module’s required field watches. --reg installs a connected listener and starts those watches, so keep a registration active while relying on policy notifications.

--reg is a connection-bound listener, not a policy selector. It always subscribes to all seven conditions in the table, regardless of any condition options also present on the command line. Stop it with a signal; the stored policy itself remains in the host engine.

Legacy examples#

Display the policy for group 2:

$ dcgmi policy --get --group 2

Store the reset and short-validation selections for double-bit ECC or PCIe errors. DCGM reports but does not execute these selections:

$ dcgmi policy --set 1,1 --group 2 --eccerrors --pcierrors

Notify without taking an action when temperature reaches 85 degrees Celsius:

$ dcgmi policy --set 0,0 --group 2 --maxtemp 85

Wait for policy notifications:

$ dcgmi policy --reg --group 2