nv-hostengine#

NAME#

nv-hostengine - run the standalone NVIDIA DCGM host engine

SYNOPSIS#

nv-hostengine [--no-daemon] [--port <port>]
              [--bind-interface <address>] [<start-options>]
nv-hostengine [--no-daemon] [--domain-socket [<path>]]
              [<start-options>]
nv-hostengine [--no-daemon] [--port <port>]
              [--vsock-cid [<cid>]] [<start-options>]
nv-hostengine --term [--pid <path>]
nv-hostengine --version
nv-hostengine --help

<start-options> are --pid, --log-level, --log-filename, --log-rotate, --denylist-modules, --service-account, and --home-dir.

DESCRIPTION#

nv-hostengine loads the DCGM shared library into a persistent standalone process, discovers supported local entities, maintains watches and management state, and accepts requests from dcgmi and DCGM API clients.

With no options, it daemonizes and opens a TCP listener on 127.0.0.1:5555. Exactly one of TCP, Unix-domain socket, or VSOCK transport is selected for a process. Most production installations run the binary in the foreground under nvidia-dcgm.service so the service manager owns its lifecycle.

The binary can run without root privileges, but prints a warning and cannot provide features for which the host-engine user lacks device or operating- system permissions. Configuration management and portions of diagnostics can require root. On DGX-2 and HGX-2 systems, the NVSwitch module requires nv-hostengine to run as root. Fabric Manager is a separate service; see NVSwitch and ConnectX.

Note

Root versus non-root execution changes the process credentials used for GPU devices, files, and child executables; it does not merely change the command prompt. See the process-credentials note and the privilege note.

OPTIONS#

Process operation#

-n, --no-daemon

Remain in the foreground instead of double-forking. Use this mode under systemd or another process supervisor. Foreground mode does not create, read, or enforce the PID file selected by --pid.

-t, --term

Read a PID from --pid, send that process SIGTERM, and wait up to 30 seconds for it to disappear. This is for a self-daemonized host engine that created the same PID file. It does not locate a listener by port or socket. Other startup options are accepted with --term but have no effect.

--pid <path>

Select the PID file used by self-daemonized startup and --term. The default is /var/run/nvhostengine.pid. The file is created only after successful daemon initialization. A foreground --no-daemon process neither creates it nor checks it for another instance.

Transport selection#

-p <port>, --port <port>

Select the TCP or VSOCK port. The default is 5555. The parser accepts an unsigned 16-bit value from 0 through 65535; use 1 through 65535. Port 0 asks the kernel for an ephemeral port, but the startup message still reports 0, so clients cannot discover the selected port from nv-hostengine output. This option has no effect on Unix-domain socket transport.

-b <address>, --bind-interface <address>

Use TCP and bind to one numeric IPv4 or IPv6 address. IPv6 values can be bracketed, for example [::1]. ALL or all binds an IPv6 wildcard socket with IPv4-mapped connections enabled. The default is 127.0.0.1. Host names and mixed-case forms such as All are not resolved as aliases.

-d [<path>], --domain-socket [<path>]

Use a Unix-domain socket and open no TCP listener. If the option has no value, the path is /tmp/nv-hostengine. Use an absolute path no longer than the platform’s sockaddr_un.sun_path capacity, normally 107 bytes plus the terminating null.

Immediately before binding, DCGM calls unlink() on the supplied path without first confirming that it names a socket. Reserve the path solely for this listener. The socket node can remain after shutdown and is removed by the next start using that path.

-c [<cid>], --vsock-cid [<cid>]

Use VSOCK. With no value, bind to any local VSOCK context ID. With a value, use a decimal unsigned context ID. --port selects the VSOCK port.

--bind-interface, --domain-socket, and --vsock-cid are mutually exclusive. Omitting all three selects TCP on the loopback address.

Logging#

--log-level <level>

Set the file/base logger threshold. Accepted values are NONE, FATAL, ERROR, WARN, INFO, DEBUG, and VERB, matched without regard to case. The default is ERROR. DCGM 4.6 does not reject an unrecognized string; it treats it as unspecified and falls back to the environment or default, so use one of the listed values exactly.

-f <path>, --log-filename <path>

Write the host-engine log to <path>. The default is /var/log/nv-hostengine.log. A value of - routes the file logger to standard output. Use an absolute path, particularly in daemon and systemd configurations.

--log-rotate

Accepted by the DCGM 4.6 parser, but not read by the startup implementation. Its presence does not change logger initialization or file handling. Configure log rotation externally until the implementation is connected to this option.

Modules and child execution#

--denylist-modules <id-list>

Prevent the comma-separated numeric module IDs from loading. Valid IDs are 1 through 10 as listed in MODULE ID REFERENCE. Core module 0 is required and rejected. Repeated IDs are collapsed. The denylist is fixed before clients connect and lasts for this host-engine process.

--service-account <username>

Select the local account used for supported unprivileged child processes, including diagnostic and multi-node diagnostic processes. This option does not change the UID of nv-hostengine itself and does not authenticate clients. With no option, children inherit the host-engine identity.

Startup stores the string without resolving the account. A nonexistent or inaccessible account can therefore allow the listener to start and fail only when a child process is launched. Use an account returned by getent passwd <username> and grant it access to required binaries, libraries, directories, GPUs, and SSH credentials.

See service and child-process identities for how Linux credentials and executable access apply to the packaged service.

--home-dir <path>

Override DCGM_HOME_DIR, attempt to create the directory if it does not exist, and change the host engine’s working directory to it before logger and listener initialization. The parent directory must already exist; creation is not recursive. Failure to create or enter the directory aborts initialization.

The directory is the default location for diagnostic logs and statistics. It does not replace the absolute default host-engine log path. Relative log, socket, and daemon PID paths can be resolved after this working-directory change; use absolute paths for all three.

Information and parser options#

--version

Print local build version, ID, date, type, source commit, architecture, platform, and CRC, then exit without initializing DCGM.

-h, --help

Print generated usage information and exit without initializing DCGM.

--, --ignore_rest

Stop parsing labeled options. This is an internal TCLAP compatibility feature and is not needed in normal use.

TRANSPORT AND CLIENT ADDRESSES#

Listener and client address forms#

Transport

Listener selection

dcgmi <command> --host address

TCP loopback

Default, or --bind-interface 127.0.0.1 --port 5555

127.0.0.1 or 127.0.0.1:5555

TCP IPv6

--bind-interface '[::1]' --port 5555

tcp://[::1]:5555

Unix-domain socket

--domain-socket /absolute/path

unix:///absolute/path

VSOCK

--vsock-cid <cid> --port 5555

vsock://<cid>:5555

The TCP and VSOCK listeners do not expose command-line options for TLS or general client authentication. Treat the endpoint as an administrative interface. Keep the default loopback binding unless remote access is required, and protect non-loopback listeners with host and network access controls.

Unix-socket ownership and mode come from the host-engine user, containing directory, and process umask; DCGM does not apply a requested owner or mode. In daemon mode nv-hostengine sets umask 0 before binding. In foreground mode it inherits the supervisor’s umask. Configure the directory and service umask for the clients that require access.

Note

Binding TCP beyond loopback can make the administrative listener reachable through host interfaces, subject to routing and firewall rules; Unix sockets remain local and use filesystem access controls. See ip(7), tcp(7), and unix(7); for host-firewall policy, consult the official Ubuntu firewall guide or upstream firewalld port and service guide as appropriate.

Use a client library from the same DCGM release where practical. Compare nv-hostengine --version with the client and host-engine information from dcgmi -v when investigating protocol or feature mismatches.

MODULE ID REFERENCE#

IDs accepted by --denylist-modules#

ID

Module

Function removed when denylisted

1

NvSwitch

NVSwitch management

2

VGPU

Virtual GPU management

3

Introspection

Host-engine resource introspection

4

Health

Background health monitoring

5

Policy

Policy management and notifications

6

Config

GPU configuration management

7

Diag

Local active diagnostics

8

Profiling

Profiling metric collection

9

SysMon

NVIDIA CPU and system monitoring

10

MnDiag

Multi-node diagnostics

Modules normally load on first use. Verify the effective state with dcgmi modules --list; see dcgmi modules and Modules. A startup denylist must be changed by restarting the host engine with different arguments.

DAEMON AND SYSTEMD MODE#

Self-daemonized mode uses the selected PID file as its only single-instance guard. A live PID in that file prevents startup even if the process is unrelated; --term likewise signals the recorded PID without verifying its executable. Use a dedicated, root-controlled absolute path. A stale file is removed on the next daemonized start after its PID is no longer present.

The packaged DCGM 4.6 unit on Linux uses this configuration:

[Service]
User=root
PrivateTmp=false
Environment="DCGM_HOME_DIR=/var/log/nvidia-dcgm"
ExecStart=/usr/bin/nv-hostengine -n --service-account nvidia-dcgm
Restart=on-abort

Because -n is present, systemd tracks the real host-engine process and no /var/run/nvhostengine.pid is created. Use systemctl stop, restart, or a normal service-manager signal for that process; nv-hostengine --term cannot stop it through the absent default PID file. Put local changes in a systemd override rather than editing the vendor unit under /usr/lib; see Customizing the systemd Service.

Do not omit -n from a foreground service unit. In self-daemonized mode the process launched by systemd exits after initialization and the actual server is its double-forked descendant.

HOST-ENGINE STATE AND RESTARTS#

Groups, field groups, watches, retained samples, jobs, health and policy configuration, profiling pause state, module state, and DCGM configuration targets are in-memory objects owned by the running host engine. A restart disconnects clients, invalidates process-local IDs and handles, and loses that state. Workflows that require it must recreate it after readiness is restored.

Restarting does not necessarily undo effects already applied outside the process. For example, a GPU configuration value that was successfully written can remain on the GPU even though DCGM has lost the target it would use to enforce that value. Inspect actual device state after a restart instead of assuming either rollback or reapplication.

Two host-engine processes on different endpoints maintain independent state while managing the same hardware and can issue conflicting watches or management operations. Run one supervised standalone instance per node unless an isolated test explicitly requires another.

ENVIRONMENT#

DCGM_HOME_DIR

Working directory and default diagnostic-output directory. --home-dir takes precedence. The packaged systemd unit sets /var/log/nvidia-dcgm.

__DCGM_DBG_FILE

Host-engine log destination used when --log-filename is omitted. The command-line option takes precedence; the final fallback is /var/log/nv-hostengine.log.

__DCGM_DBG_LVL

Host-engine logging severity used when --log-level is omitted. Use one of the severity names listed above. The command-line option takes precedence; the final fallback is ERROR.

CUDA_VISIBLE_DEVICES

When set in the host-engine environment before startup, limits the GPUs visible to the DCGM process. Setting it only in a later dcgmi client does not change host-engine inventory.

DCGM_HANGDETECT_DISABLE, DCGM_HANGDETECT_EXPIRY_SEC, and DCGM_HANGDETECT_TERMINATE

Control host-engine hang detection, which is available in DCGM 4.4.2 and later. It monitors selected host-engine and module threads and processes; the availability boundary does not imply that every internal task is monitored. Coverage of the dedicated NVML task-runner thread was added in DCGM 4.5.0. See Host Engine Environment Variables Affecting Hang Detection for exact behavior.

Host-engine hang detection is independent of diagnostic task hang detection, which is available in DCGM 4.4 and later and uses the NVVS_HANGDETECT_DISABLE and NVVS_HANGDETECT_EXPIRY_SEC variables. See Environment.

Diagnostic plugins and subprocesses inherit the service environment. Paths to nvvs, plugins, external diagnostic executables, OpenMPI, and required shared libraries must therefore be configured on nv-hostengine, not only in the shell that runs dcgmi. See Environment and Prerequisites.

For the Linux inheritance model behind this behavior, see process environments and, when library search paths are involved, shared-library loading.

SIGNALS#

SIGHUP, SIGINT, SIGQUIT, and SIGTERM all request the same orderly shutdown. SIGHUP does not reload options or environment. A normal handled shutdown stops the embedded DCGM engine and exits successfully. SIGPIPE is ignored so a disconnected client does not terminate the server.

For background on signal delivery and the shell status produced by a signal, see signal(7) and EXIT STATUS.

EXIT STATUS#

0 is returned for help, version information, a successful --term, and a normal handled shutdown. A generated-parser error exits with 1. A post-parse option conflict returns -1 from main(), which a POSIX shell reports as 255. A failed --term exits with 1.

DCGM 4.6 has two startup-status exceptions that matter to automation:

  • In self-daemonized mode, the original process exits with status 0 whenever the child signals that console output can close. The child sends the same signal on initialization failure, so status 0 does not prove that a listener or daemon remains.

  • In --no-daemon mode, an initialization failure enters the same cleanup path and sends SIGUSR1 to the process itself. A POSIX shell commonly reports this as 138 (128 + signal 10), rather than a normal failure code.

After startup, verify both the supervised process/listener and a real client request, for example dcgmi discovery --host <address> --list. Also inspect the host-engine log; do not use the launch command’s status alone as a readiness check.

FILES#

/usr/lib/systemd/system/nvidia-dcgm.service

Packaged systemd unit.

/var/run/nvhostengine.pid

Default self-daemon PID file. Not created by --no-daemon.

/var/log/nv-hostengine.log

Default host-engine log.

/var/log/nvidia-dcgm

Packaged service’s diagnostic home and default diagnostic-output directory.

/tmp/nv-hostengine

Default socket path when --domain-socket has no value.

EXAMPLES#

Run in the foreground on the default loopback TCP endpoint:

# nv-hostengine --no-daemon --service-account nvidia-dcgm \
    --home-dir /var/log/nvidia-dcgm

Run in the foreground on a Unix-domain socket:

# nv-hostengine --no-daemon \
    --domain-socket /run/nvidia-dcgm/nv-hostengine.sock \
    --service-account nvidia-dcgm

Connect a client to that socket:

$ dcgmi discovery --host unix:///run/nvidia-dcgm/nv-hostengine.sock --list

Start a self-daemonized instance with an explicit PID file, then terminate that same instance:

# nv-hostengine --pid /run/nvhostengine.pid
# nv-hostengine --term --pid /run/nvhostengine.pid

Prevent the VGPU, Introspection, and Profiling modules from loading:

# nv-hostengine --no-daemon --denylist-modules 2,3,8

SEE ALSO#

dcgmi, dcgmi modules, dcgmi set, dcgmi diag, dcgmi mndiag, Modules, Install DCGM, Learn, Customizing the systemd Service, Debugging and Troubleshooting