Troubleshooting
NVIDIA Fleet Intelligence (NFI) has two components:
- The agent
- The service
Since the service is hosted by NVIDIA, most troubleshooting will occur at the agent, which is installed on one or many nodes within the customer environment.
Often the nodes are similar or identical, so it may be possible to debug a single representative node, but nodes may be different, so “representative nodes” may be difficult to identify and may vary based on network, location, data center, and so on.
There are two main methods to install the NFI agent:
- a base operating system install (for example, Red Hat or Debian/Ubuntu), and
- a Helm installation of the agent as a DaemonSet within Kubernetes.
The triage will be slightly different between the two installation methods.
Agent Troubleshooting
For base OS installation problems, it is important that a CUDA repository has been enabled, as the Fleet Intelligence agent installation will pull in required dependencies (Ubuntu and RedHat), including Data Center GPU Manager (DCGM) and corelib.
For Helm installations, NFI requires DCGM to already be running. In Kubernetes installations, this is often accomplished through the GPU Operator. DCGM must be installed in Host Engine mode (i.e., shared mode). This shared DCGM mode allows multiple services to access it simultaneously. Please see the details in the Helm Installation.
If installation of the package is successful, the agent will start and log to the journal. To check the status of the agent service, run the following command:
To view the logs, run the following command:
Confirm that the GPUs are present and the driver is loaded by running nvidia-smi:
Agent Enrollment Troubleshooting
Run the precheck sequence manually to confirm the configuration is correct:
Gather the machine information as the NFI agent sees it to confirm configuration:
Messages in logs like “msg”:“failed to get UUID from dmidecode, trying to read from file” are warnings; they are expected and not a problem.
Confirm connectivity:
This confirms DNS and basic connectivity, but keep in mind that ICMP (ping) may be blocked by a firewall or network configuration.
Confirm HTTPS connectivity:
This should quickly return a 404 (file not found) error, indicating that the connection was OK. If it takes a long time and times out, it indicates a problem with HTTP connectivity. Check firewall rules and network configuration to confirm that the agent can access the NFI service.