Configuring System Settings#

This section provides information about less common configuration options after a system has been installed.

Refer also to DGX OS Connectivity Requirements for a list of network ports used by various services.

Network Configuration#

This section provides information about configuring the network in your DGX system.

Configuring Network Proxies#

If your network needs to use a proxy server, you need to set up configuration files to ensure the DGX system communicates through the proxy.

Network Proxy Configuration For the OS and Most Applications#

Configure proxy environment variables in /etc/environment when the DGX system must use a proxy to reach external services. These settings are used by the operating system and most applications, including command-line tools that inherit the system environment. Configure APT and Docker separately because they can require service-specific proxy settings.

Add proxy addresses to the /etc/environment file. The proxy address should be added to the file below the PATH line. Note that the username and password are optional.

http_proxy="http://<username>:<password>@<host>:<port>/"
ftp_proxy="ftp://<username>:<password>@<host>:<port>/"
https_proxy="https://<username>:<password>@<host>:<port>/"
no_proxy="localhost,127.0.0.1,localaddress,.localdomain.com"
HTTP_PROXY="http://<username>:<password>@<host>:<port>/"
FTP_PROXY="ftp://<username>:<password>@<host>:<port>/"
HTTPS_PROXY="https://<username>:<password>@<host>:<port>/"
NO_PROXY="localhost,127.0.0.1,localaddress,.localdomain.com"

Both the uppercase and lowercase versions of the proxy environment variables must be changed as shown in the example below:

http_proxy="http://myproxy.server.com:8080/"
HTTP_PROXY="http://myproxy.server.com:8080/"

Configuring APT to Use a Proxy#

If APT must access package repositories through a proxy, configure an APT-specific proxy file. This enables APT to download package metadata and software updates when direct access to the repositories is not permitted.

Edit or create the /etc/apt/apt.conf.d/myproxy proxy configuration file and include the following lines:

Acquire::http::proxy "http://<username>:<password>@<host>:<port>/";
Acquire::ftp::proxy "ftp://<username>:<password>@<host>:<port>/";
Acquire::https::proxy "https://<username>:<password>@<host>:<port>/";

For example:

Acquire::http::proxy "http://myproxy.server.com:8080/";
Acquire::ftp::proxy "ftp://myproxy.server.com:8080/";
Acquire::https::proxy "https://myproxy.server.com:8080/";

Configuring ConnectX from InfiniBand to Ethernet#

Many DGX systems are equipped with NVIDIA ConnectX network controllers and are typically used for cluster communications. By default, the controllers are configured as InfiniBand ports. You can optionally configure the ports for Ethernet.

Before or after you reconfigure the port, make sure that the network switch that is connected to the port is also reconfigured to Ethernet or that the port is connected to a different switch configured for Ethernet.

The code samples in the following sections use the mlxconfig command.

Determining the Current Port Configuration#

Perform the following steps to determine the current configuration for the port.

  • Query the devices:

    sudo mlxconfig -e -d aa:00.0 q |grep -e 'Device\|Configurations\|LINK_TYPE'

    The following example shows the output for one of the port devices on an NVIDIA DGX B200 system. It shows the device path and the default, current, and next boot configurations, which are all set to IB(1).

    ...
Device #1:
Device type:        BlueField3
Device:             aa:00.0
Configurations:                           Default             Current         Next Boot
*       LINK_TYPE_P1                      IB(1)               IB(1)           IB(1)
*       LINK_TYPE_P2                      IB(1)               IB(1)           IB(1)

    • IB(1) indicates the port is configured for InfiniBand.

    • ETH(2) indicates the port is configured for Ethernet.

Determine the device path bus numbers for the slot number of the port that you want to configure. Refer to the following documents for more information:

Configuring the Port#

  1. Use the mlxconfig command with the set LINK_TYPE_P<x> argument for each port you want to configure.

    The following sample command sets port LINK_TYPE_P1 of the controller with PCI ID aa:00.0 to Ethernet (2):

    sudo mlxconfig -y -d aa:00.0 set LINK_TYPE_P1=2

    The following example output is from an NVIDIA DGX B200 system.

    Device #1:
----------

Device type:        BlueField3
Name:               900-9D3B6-00CN-A_Ax
Description:        NVIDIA BlueField-3 B3240 P-Series Dual-slot FHHL DPU; 400GbE / NDR IB (default mode); Dual-port QSFP112; PCIe Gen5.0 x16 with x16 PCIe extension option; 16 Arm cores; 32GB on-board DDR; integrated BMC; Crypto Enabled
Device:             aa:00.0

Configurations:                                          Next Boot       New
        LINK_TYPE_P1                                     ETH(2)          IB(1)

Apply new Configuration? (y/n) [n] : y
Applying... Done!
-I- Please reboot machine to load new configurations.

    Here is an example that sets port LINK_TYPE_P2 to Ethernet:

    sudo mlxconfig -y -d aa:00.0 set LINK_TYPE_P2=2

  2. (Optional) Run mlxconfig again to confirm the change:

    sudo mlxconfig -e -d aa:00.0 q |grep -e 'Device\|Configurations\|LINK_TYPE'

    In the following example output, port LINK_TYPE_P1 is set to ETH(2) for the next boot. The output shows the device path and the default, current, and next boot configurations.

    ...
Device #1:
Device type:        BlueField3
Device:             aa:00.0
Configurations:                           Default             Current         Next Boot
*       LINK_TYPE_P1                      IB(1)               IB(1)           ETH(2)
*       LINK_TYPE_P2                      IB(1)               IB(1)           IB(1)

  3. Power off and then power on the system for the change to take effect.

    Wait for the operating system to boot.

Configuring the ConnectX-8 Port for Splitting#

To change the ConnectX-8 interface from a single 800 Gbps InfiniBand port to two 400 Gb Ethernet ports:

  1. Set the link type for Port 1 to Ethernet (2).

    sudo mlxconfig -d <device> set LINK_TYPE_P1=2

    Replace <device> with the actual PCI device ID of your ConnectX-8 device.

  2. Set the number of planes for Port 1 to 0.

    sudo mlxconfig -d <device> set NUM_OF_PLANES_P1=0

  3. Define the physical module split configuration.

    sudo mlxconfig -d <device> set MODULE_SPLIT_M0[0..3]=1 MODULE_SPLIT_M0[4..7]=2 MODULE_SPLIT_M0[8..15]=FF

    This specific set of values maps the internal physical lanes of the 800 Gbps port into two separate 400 GbE logical ports.

  4. Set the number of Physical Functions (PFs) to 2, allowing the operating system to recognize two 400 GbE network interfaces.

    sudo mlxconfig -d <device> set NUM_OF_PF=2

  5. Power off and then power on the system for the change to take effect.

    Wait for the operating system to boot.

To revert the device interface to its original single 800 Gbps InfiniBand (IB) port configuration:

sudo mlxconfig -d <device> reset

Note

The reset action will revert all settings on the device, including any other specific settings you have modified.

For more detailed information on configuring ConnectX-8 devices, refer to Port Splitting Configurations.

Docker Configuration#

To ensure that Docker can access the NVIDIA GPU Cloud (NGC) container registry through a proxy, Docker uses environment variables.

For best practice recommendations on configuring proxy environment variables for Docker, refer to systemd unit file.

Preparing the DGX System to be Used With Docker#

Some initial setup of the DGX system is required to ensure that users have the required privileges to run Docker containers and to prevent IP address conflicts between Docker and the DGX system.

Enabling Users To Run Docker Containers#

By default, the Docker daemon socket is owned by root, so users must use sudo to run Docker commands. Alternatively, you can add a user to the docker group. Members of this group can run Docker commands without sudo access; because the Docker daemon runs as root, members of the docker group can effectively obtain root privileges on the host through Docker.

Add only trusted users to the docker group. Before allowing multiple users to run Docker commands, consult your IT department to determine whether this complies with your organization’s security policies. For the security implications, see Docker daemon attack surface.

To add an existing user to the docker group, run this command:

sudo usermod -aG docker user-login-id

Where user-login-id is the login ID of the existing user that you are adding to the docker group.

Configuring Docker IP Addresses#

To ensure that your DGX system can access the network interfaces for Docker containers, Docker should be configured to use a subnet distinct from other network resources used by the DGX system.

By default, Docker uses the 172.17.0.0/16 subnet. Consult your network administrator to find out which IP addresses are used by your network. If your network does not conflict with the default Docker IP address range, no changes are needed and you can skip this section. However, if your network uses the addresses in this range for the DGX system, you should change the default Docker network addresses.

You can change the default Docker network addresses by editing the /etc/docker/daemon.json file. If the file already contains settings, add the following keys to its existing JSON object.

  1. Open the daemon.json file for editing.

    sudo vi /etc/docker/daemon.json

  2. Add the following settings, replacing the example bridge IP address and IP address range with values for your network.

    Consult your IT administrator for the correct addresses.

    {
  "bip": "192.168.127.1/24",
  "fixed-cidr": "192.168.127.128/25"
}

  3. Save and close the /etc/docker/daemon.json file.

  4. Restart Docker.

    sudo systemctl restart docker

Connectivity Requirements for NGC Containers#

To run NVIDIA NGC containers from the NGC container registry, your network must be able to access the following URLs:

For x86_64 systems:

For ARM64 systems:

To verify connection to nvcr.io, run the following command:

wget https://nvcr.io/v2

Connection and Transport Layer Security (TLS) verification confirms that the system reached the NGC Container Registry through the configured network path. Because this request does not provide NGC credentials, a 401 Unauthorized response is expected:

--2018-08-01 19:42:58-- https://nvcr.io/v2
Resolving nvcr.io (nvcr.io) --> 52.8.131.152, 52.9.8.8
Connecting to nvcr.io (nvcr.io)|52.8.131.152|:443. --> connected.
HTTP request sent, awaiting response. --> 401 Unauthorized

Configuring Static IP Addresses for the Network Ports#

Here are the steps to configure static IP addresses for network ports.

During the initial boot setup process for your DGX system, one of the steps was to configure static IP addresses for a network interface. If you did not configure the addresses at that time, you can configure the static IP addresses from the Ubuntu command line using the following instructions.

Note

If you are connecting to the DGX console remotely, connect by using the BMC remote console. If you connect using SSH, your connection will be lost when you complete the final step. Also, if you encounter issues with the configuration file, the BMC connection will help with troubleshooting.

If you cannot remotely access the DGX system, connect a display with a 1440x900 or lower resolution, and a keyboard directly to the DGX system.

  1. Determine the port designation that you want to configure, based on the physical Ethernet port that you have connected to your network.

    List the available network interfaces:

    ip link show

    Identify the interface that corresponds to the connected physical port.

  2. Identify the Netplan YAML file that configures the interface, then edit that file. Netplan configuration file names vary by installation.

    ls /etc/netplan/

    Note

    Use spaces, not tabs, and adapt the following sample to your existing Netplan configuration.

    # This file describes the network interfaces available on your system
# For more information, see netplan(5).

network:
  version: 2
  renderer: networkd
  ethernets:
    enp226s0:
      dhcp4: no
      addresses:
        - 10.10.10.2/24
      routes:
        - to: default
          via: 10.10.10.1
      nameservers:
        addresses: [ 8.8.8.8 ]

    Consult your network administrator for your site-specific values such as network, gateway, and nameserver addresses. Replace enp226s0 with the interface designation that you determined in the preceding step.

  3. Save the file.

  4. Apply the changes.

    sudo netplan apply

Note

If the command does not return to the command prompt, run sudo netplan --debug apply to collect diagnostic output. For more information, refer to the netplan apply documentation.

Managing CPU Mitigations#

DGX OS software includes security updates to mitigate CPU speculative side-channel vulnerabilities. These mitigations can decrease the performance of deep learning and machine learning workloads.

If your DGX system installation incorporates other measures to mitigate these vulnerabilities, such as measures at the cluster level, you can disable the CPU mitigations for individual DGX nodes and increase performance.

Determining the CPU Mitigation State of the DGX System#

Here is information about how you can determine the CPU mitigation state of your DGX system.

To determine whether the nvidia-no-mitigations profile has disabled CPU mitigations, check the kernel command line.

cat /proc/cmdline

If the output includes mitigations=off, CPU mitigations are disabled. If it does not, the kernel uses its configured CPU mitigations.

To see the status of individual vulnerabilities, review files in the /sys/devices/system/cpu/vulnerabilities/ directory. Because the output can contain both Mitigation and Vulnerable entries, do not use it alone to determine whether mitigations=off is active.

Disabling CPU Mitigations#

Caution

Performing the following instructions will disable the CPU mitigations provided by the DGX OS software.

To disable CPU mitigations, append the nvidia-no-mitigations profile to your currently active tuned configuration.

  1. Check your system’s active profiles:

    $ tuned-adm active
Current active profile: <active_prof_1> ... <active_prof_N>

  2. Append the nvidia-no-mitigations profile to the existing active profiles:

    $ sudo tuned-adm profile <active_prof_1> ... <active_prof_N> nvidia-no-mitigations

  3. Reboot the system to apply the changes:

    $ sudo reboot

  4. Verify that the CPU mitigations are disabled:

    $ cat /proc/cmdline

    Confirm that the output includes mitigations=off.

Re-enabling CPU Mitigations#

To re-enable CPU security mitigations, remove the nvidia-no-mitigations profile from your active tuned configuration.

  1. Check your system’s active profiles:

    $ tuned-adm active
Current active profile: <active_prof_1> ... <active_prof_N> nvidia-no-mitigations

  2. Apply the active profiles without the nvidia-no-mitigations profile:

    $ sudo tuned-adm profile <active_prof_1> ... <active_prof_N>

  3. Reboot the system to apply the changes:

    $ sudo reboot

  4. Verify that the CPU mitigations are enabled:

    $ cat /proc/cmdline

    Confirm that the output does not include mitigations=off.

Performance Impact of Enabling init_on_alloc#

The CONFIG_INIT_ON_ALLOC_DEFAULT_ON Linux kernel configuration option controls whether the kernel fills newly allocated pages and heap objects with zeroes by default. You can control this setting with the init_on_alloc=<0|1> kernel parameter.

Although init_on_alloc=1 is an Ubuntu kernel-hardening default, the BaseOS 9 nvidia-base TuneD profile sets init_on_alloc=0 for DGX systems. This avoids the performance cost of zeroing every newly allocated buffer page.

Enabling init_on_alloc can reduce network interface controller performance on DGX systems. The impact is more likely with the inbox OFED driver than with the DOCA-OFED driver because the DOCA-OFED driver allocates a larger page cache that better tolerates the cost of zeroing pages.

If your deployment requires the additional hardening that init_on_alloc=1 provides, evaluate the performance impact before overriding the DGX OS default.

  1. Identify the active TuneD profiles.

    $ tuned-adm active
Current active profile: <active_prof_1> ... <active_prof_N>

  2. Edit the tuned.conf file for the active performance profile in /usr/lib/tuned/profiles/<active-performance-profile>/. Add or change the following entry in the [bootloader] section:

    cmdline_init_on_alloc=init_on_alloc=1

  3. Reapply the active TuneD profiles.

    $ sudo tuned-adm profile <active_prof_1> ... <active_prof_N>

  4. Reboot the system to apply the change.

    sudo reboot

  5. Verify that the change took effect.

    cat /proc/cmdline

    Confirm that the output includes init_on_alloc=1.

Managing the DGX Crash Dump Feature#

This section provides information about managing the DGX Crash Dump feature. You can use the script that is included in the DGX OS to manage this feature.

Using the Script#

Here are commands that help you complete the necessary tasks with the script.

  • To enable only dmesg crash dumps, run:

    /usr/sbin/nvidia-kdump-config enable-dmesg-dump

    This option reserves memory for the crash kernel.

  • To enable both dmesg and vmcore crash dumps, run:

    /usr/sbin/nvidia-kdump-config enable-vmcore-dump

    This option reserves memory for the crash kernel.

  • To disable crash dumps, run:

    /usr/sbin/nvidia-kdump-config disable

This option disables the use of kdump and ensures that no memory is reserved for the crash kernel.

Connecting to Serial Over LAN#

You can connect to serial over LAN.

Note

This feature applies only to DGX systems that have the BMC.

While dumping vmcore, the BMC screen console goes blank approximately 11 minutes after the crash dump is started. To view the console output during the crash dump, connect to serial over LAN as follows:

ipmitool -I lanplus -H <BMC-IP> -U <username> -P <password> sol activate

Filesystem Quotas#

Here is some information about filesystem quotas.

When running NGC containers, you might need to limit the amount of disk space that is used on a filesystem to avoid filling up the partition. Refer to How to Set Filesystem Quotas on Ubuntu 18.04 about how to set filesystem quotas on Ubuntu 18.04 and later.

Running Workloads on Systems with Mixed Types of GPUs#

DGX Station systems include NVIDIA GPUs for compute workloads and a DGX Display GPU that drives the attached monitor. Use the NVIDIA GPUs for high-performance and AI workloads; use the DGX Display GPU for display output.

For compute-intensive applications and workloads, use NVIDIA GPUs rather than the DGX Display GPU. The DGX Display GPU is intended for display output and does not provide the direct NVLink connectivity, memory capacity, or performance characteristics of the installed NVIDIA GPUs. Verify that the application or workload selects the NVIDIA GPUs.

When running applications on a DGX Station system, select the appropriate NVIDIA GPUs to obtain the intended performance. You can select GPUs in the following ways:

To list the NVIDIA GPUs and DGX Display GPU available on the system, run the nvidia-smi --list-gpus command:

nvidia-smi --list-gpus
GPU 0: Graphics Device (UUID: GPU-269d95f8-328a-08a7-5985-ab09e6e2b751)
GPU 1: Graphics Device (UUID: GPU-0f2dff15-7c85-4320-da52-d3d54755d182)
GPU 2: Graphics Device (UUID: GPU-dc598de6-dd4d-2f43-549f-f7b4847865a5)
GPU 3: DGX Display (UUID: GPU-91b9d8c8-e2b9-6264-99e0-b47351964c52)
GPU 4: Graphics Device (UUID: GPU-e32263f2-ae07-f1db-37dc-17d1169b09bf)

Running GPU-Enabled Containers with Docker#

This section describes how to run GPU-enabled containers on a DGX system with Docker and the NVIDIA Container Toolkit. Use these procedures after installing and configuring Docker and the NVIDIA Container Toolkit and, when required, authenticating with the NVIDIA NGC Container Registry.

Note

For more information about using docker run to make NVIDIA GPUs available to a container, refer to GPU access in the Docker documentation.

To allow Docker to use the system’s NVIDIA GPUs, specify the --gpus option when you execute docker run to create and start a container. The --gpus option controls which NVIDIA GPUs are available to the container. Include --gpus all to make all available NVIDIA GPUs accessible, --gpus n to make n GPUs accessible, or --gpus '"device=<GPU-list>"' to select specific GPUs.

To verify that Docker exposes all available GPUs to a container, start a test container with docker run, specify --gpus all, and run nvidia-smi --list-gpus in the container:

The following output lists the GPUs available to the container:

docker run --gpus all --rm -it ubuntu nvidia-smi --list-gpus
GPU 0: Graphics Device (UUID: GPU-269d95f8-328a-08a7-5985-ab09e6e2b751)
GPU 1: Graphics Device (UUID: GPU-0f2dff15-7c85-4320-da52-d3d54755d182)
GPU 2: Graphics Device (UUID: GPU-dc598de6-dd4d-2f43-549f-f7b4847865a5)
GPU 3: Graphics Device (UUID: GPU-e32263f2-ae07-f1db-37dc-17d1169b09bf)

To verify that Docker exposes a specified number of GPUs to a container, start a test container with docker run, specify --gpus n, and run nvidia-smi --list-gpus in the container. On the example system, n can be 1, 2, 3, or 4:

docker run --gpus 2 --rm -it ubuntu nvidia-smi --list-gpus
GPU 0: Graphics Device (UUID: GPU-269d95f8-328a-08a7-5985-ab09e6e2b751)
GPU 1: Graphics Device (UUID: GPU-0f2dff15-7c85-4320-da52-d3d54755d182)

To verify a specific GPU selection, start a test container with docker run, specify --gpus '"device=<GPU-list>"', and run nvidia-smi --list-gpus in the container. Confirm that the output lists only the selected GPU UUIDs; GPU indexes can be renumbered inside the container. For example, --gpus all lists all available GPUs, whereas a device selection lists only the specified GPUs.

docker run --gpus '"device=GPU-dc598de6-dd4d-2f43-549f-f7b4847865a5,GPU-e32263f2-ae07-f1db-37dc-17d1169b09bf"' --rm -it ubuntu nvidia-smi --list-gpus
GPU 0: Graphics Device (UUID: GPU-dc598de6-dd4d-2f43-549f-f7b4847865a5)
GPU 1: Graphics Device (UUID: GPU-e32263f2-ae07-f1db-37dc-17d1169b09bf)

Running CUDA Workloads on NVIDIA GPUs on Bare Metal#

This section describes how to select NVIDIA GPUs for CUDA workloads that run directly on the system. To control GPU visibility for a CUDA application, set CUDA_VISIBLE_DEVICES in the environment from which you start the application. This variable determines which GPUs the application and its child processes can use. When you specify GPU ordinals rather than UUIDs, CUDA enumerates available GPUs by using the FASTEST_FIRST heuristic by default. Set CUDA_DEVICE_ORDER=PCI_BUS_ID to enumerate GPUs by ascending PCI bus ID. If repeatable GPU selection is important, use GPU UUIDs with CUDA_VISIBLE_DEVICES rather than device ordinals.

Set these variables before starting the application. They apply only to the application process and its child processes; they do not change the system-wide GPU configuration. You can set them for one application launch by prefixing its command, for example:

CUDA_VISIBLE_DEVICES=GPU-<UUID> CUDA_DEVICE_ORDER=PCI_BUS_ID <application>

For more information about these CUDA environment variables to select NVIDIA GPUs for CUDA workloads, refer to CUDA Environment Variables in the CUDA Programming Guide.

NVIDIA provides the open-source nvbandwidth utility to measure bandwidth for GPU memory-copy patterns across the available NVIDIA GPU interconnects. For information about downloading, building, and running the utility, refer to NVIDIA/nvbandwidth.

Using Multi-Instance GPU (MIG) Devices#

Multi-Instance GPU (MIG) technology is supported on compatible NVIDIA GPUs. When MIG mode is enabled and the GPU is partitioned, each MIG device can be selected independently for a containerized or bare-metal CUDA workload. For instructions about configuring and using MIG, refer to the NVIDIA Multi-Instance GPU User Guide.

Identify the MIG devices that you want to make available to a workload. The following output is from a system with GPU 0 partitioned into seven MIG devices:

nvidia-smi --list-gpus

GPU 0: Graphics Device (UUID: GPU-269d95f8-328a-08a7-5985-ab09e6e2b751)
  MIG 1g.10gb Device 0: (UUID: MIG-GPU-269d95f8-328a-08a7-5985-ab09e6e2b751/7/0)
  MIG 1g.10gb Device 1: (UUID: MIG-GPU-269d95f8-328a-08a7-5985-ab09e6e2b751/8/0)
  MIG 1g.10gb Device 2: (UUID: MIG-GPU-269d95f8-328a-08a7-5985-ab09e6e2b751/9/0)
  MIG 1g.10gb Device 3: (UUID: MIG-GPU-269d95f8-328a-08a7-5985-ab09e6e2b751/11/0)
  MIG 1g.10gb Device 4: (UUID: MIG-GPU-269d95f8-328a-08a7-5985-ab09e6e2b751/12/0)
  MIG 1g.10gb Device 5: (UUID: MIG-GPU-269d95f8-328a-08a7-5985-ab09e6e2b751/13/0)
  MIG 1g.10gb Device 6: (UUID: MIG-GPU-269d95f8-328a-08a7-5985-ab09e6e2b751/14/0)
GPU 1: Graphics Device (UUID: GPU-0f2dff15-7c85-4320-da52-d3d54755d182)
GPU 2: Graphics Device (UUID: GPU-dc598de6-dd4d-2f43-549f-f7b4847865a5)
GPU 3: DGX Display (UUID: GPU-91b9d8c8-e2b9-6264-99e0-b47351964c52)
GPU 4: Graphics Device (UUID: GPU-e32263f2-ae07-f1db-37dc-17d1169b09bf)

Copy the UUID of the MIG device that you want to make available to the workload.

To make a specific MIG device available to a Docker container, use its MIG UUID with the device value of the --gpus option. The following example makes the MIG device on GPU 0 with Device 0 available to the container being started:

docker run \
   --gpus '"device=MIG-GPU-269d95f8-328a-08a7-5985-ab09e6e2b751/7/0"' \
   --rm -it ubuntu nvidia-smi --list-gpus

GPU 0: Graphics Device (UUID: GPU-269d95f8-328a-08a7-5985-ab09e6e2b751)
  MIG 1g.10gb Device 0: (UUID: MIG-GPU-269d95f8-328a-08a7-5985-ab09e6e2b751/7/0)

To make a specific MIG device available to a bare-metal CUDA application, set CUDA_VISIBLE_DEVICES to its MIG UUID before starting the application. An example of how to make a MIG device available to a CUDA application would be to, first, build the deviceQuery application from the CUDA Samples repository. Second, start the deviceQuery application with the CUDA_VISIBLE_DEVICES environment variable set to the desired MIG UUID as shown in the next example. Output from deviceQuery should show only the selected MIG device as the CUDA device available.

CUDA_VISIBLE_DEVICES=MIG-GPU-269d95f8-328a-08a7-5985-ab09e6e2b751/7/0 ./deviceQuery

The application output should show only the selected MIG device.

Note

On DGX Station systems, nv-docker-gpus prevents Docker and containerd from accessing the system display GPU. After changing the GPU or MIG topology, restart nv-docker-gpus and Docker to update the GPUs available to containers. This service is not used on DGX server systems.

sudo systemctl restart nv-docker-gpus
sudo systemctl restart docker

Refreshing Container Device Permissions After a MIG Configuration Change#

On DGX Station systems, nv-docker-gpus creates a systemd drop-in for containerd in the /etc/systemd/system/containerd.service.d/ directory. The drop-in applies device permissions to containerd so that container workloads cannot access the system display GPU. After you create, remove, or reconfigure MIG devices, refresh the drop-in so that its DeviceAllow entries include the required NVIDIA device nodes.

Note

DGX Station support will be available in a future DGX OS release.

The following is an example of a containerd drop-in:

[Service]
DeviceAllow=/dev/nvidia1
DeviceAllow=/dev/nvidia2
DeviceAllow=/dev/nvidia3
DeviceAllow=/dev/nvidia4
DeviceAllow=/dev/nvidia-caps/nvidia-cap1
DeviceAllow=/dev/nvidia-caps/nvidia-cap2
DeviceAllow=/dev/nvidiactl
DeviceAllow=/dev/nvidia-modeset
DeviceAllow=/dev/nvidia-uvm
DeviceAllow=/dev/nvidia-uvm-tools

To refresh the drop-in after a MIG configuration change, run:

sudo systemctl restart nv-docker-gpus
sudo systemctl restart docker

Then verify that the new MIG devices are available and that a container can access a selected device:

  1. List the current MIG devices and copy the UUID of the device that you want to test:

    nvidia-smi --list-gpus

  2. Confirm that a container can access the selected MIG device:

    docker run \
   --gpus '"device=<MIG-UUID>"' \
   --rm -it ubuntu nvidia-smi --list-gpus

    The output should show only the selected MIG device.

Data Storage Configuration#

By default, DGX systems include data drives configured as a RAID 0 array mounted at /raid. RAID 0 can provide high local storage capacity and throughput, but it does not provide redundancy: if any drive in the array fails, data on the array can be lost.

Use this storage for cache data, temporary files, or data that can be recreated from another source. Do not store critical, persistent, or long-term data on this array unless it is backed up elsewhere. For persistent data, use organization-managed storage, such as an NFS export.

Using the Data Drives for NFS Caching#

The local /raid array can be used as an FS-Cache backing store for an NFS mount. FS-Cache stores read-only NFS data pages locally, which can improve access to repeatedly read data.

DGX OS configures cachefilesd through nvidia-tuned-profiles to use the /raid partition. Before you configure an NFS mount for caching, ensure that:

  • The DGX system has access to an NFS server export that contains the data to cache.

  • Network routing, DNS resolution when applicable, and NFS access permissions allow the DGX system to mount the export.

  • The /raid partition is mounted and has sufficient free space for the cache.

Configuring NFS Caching with cachefilesd#

To enable local FS-Cache storage for an NFS mount, add the fsc mount option to the NFS entry in /etc/fstab.

  1. Create a mount point for the NFS export. For example:

    sudo mkdir -p /mnt/dgx-nfs-cache

  2. Edit the /etc/fstab file:

    sudoedit /etc/fstab

  3. Add an NFS entry to /etc/fstab. For example:

    <nfs-server>:<export-path> /mnt/dgx-nfs-cache nfs _netdev,fsc,nofail 0 0

    Replace <nfs-server> and <export-path> with values provided by your storage administrator. The fsc option enables FS-Cache for this mount. Add site-specific NFS protocol, security, or performance options only when required by your NFS server or network configuration.

    Note

    Evaluate FS-Cache carefully for shared data that changes frequently. Its benefits are greatest for data that is read repeatedly. The fsc option enables local caching of read-only NFS data pages. For details, refer to the NFS mount-option documentation.

  4. Mount the export:

    sudo mount /mnt/dgx-nfs-cache

  5. Verify that the cachefilesd service is active and that FS-Cache is enabled for the NFS mount:

    systemctl is-active cachefilesd
cat /proc/fs/nfsfs/volumes

    In the /proc/fs/nfsfs/volumes output, find the mount and confirm that it includes FSC=yes.

The NFS export is mounted automatically after a reboot while its /etc/fstab entry remains present and the cachefilesd service is enabled.

Disabling NFS Caching with cachefilesd#

To disable FS-Cache for an NFS mount, remove the fsc option from its /etc/fstab entry, then remount the export. Stopping cachefilesd without changing the mount entry leaves the mount configured to request FS-Cache.

  1. Edit /etc/fstab and remove fsc from the NFS mount options.

  2. Unmount and remount the NFS export. For example:

    sudo umount /mnt/dgx-nfs-cache
sudo mount /mnt/dgx-nfs-cache

  3. Stop cachefilesd and prevent it from starting automatically:

    sudo systemctl disable --now cachefilesd

  4. Confirm that FS-Cache is no longer enabled for the mount:

    cat /proc/fs/nfsfs/volumes

    Confirm that the mount no longer shows FSC=yes.

Note

Disabling cachefilesd does not delete existing cached data. Do not manually modify or remove cache contents while the service is active.

Changing the RAID Level of the Data Array#

You can use the configure_raid_array.py script to change the RAID configuration of the data array.

Caution

Running configure_raid_array.py is destructive. Back up all data on the data array before you continue. It temporarily stops NVIDIA System Management (NVSM) and, if installed, cachefilesd; it unmounts /raid if it is mounted; recreates the RAID array and its ext4 file system; and then remounts /raid and restarts NVSM. For RAID 0, the script enables and starts cachefilesd. For RAID 5, it stops and disables cachefilesd.

cachefilesd stores cached NFS data locally and can generate substantial write activity. RAID 0 favors capacity and performance, which suits recreatable cache data. RAID 5 trades capacity and write performance for single-drive fault tolerance, so the script disables cachefilesd after a RAID 5 conversion to preserve that array for protected data rather than cache activity.

RAID 0 requires at least two data drives and provides no redundancy. If one SSD in the array fails, all data on the array is lost. RAID 5 requires at least three data drives, provides protection against a single drive failure, and reduces usable capacity by approximately one drive. RAID 5 is not a substitute for backups.

configure_raid_array.py. prompts for an explicit y (yes) confirmation when you use the -m option. Enter y only after you verify that the required backup is complete.

  • To change the RAID level to RAID 5, run the following command:

    sudo configure_raid_array.py -m raid5

  • To change the RAID level to RAID 0, run the following command:

    sudo configure_raid_array.py -m raid0

  • To identify the block device mounted at /raid, run:

    findmnt --noheadings --output SOURCE --target /raid

  • Confirm the RAID level of the device reported by the preceding findmnt command. For example, if it reports /dev/md0, run the mdadm command below with /dev/md0 as the argument. The mdadm command will report either Raid Level : raid0 or Raid Level : raid5.

    sudo mdadm --detail /dev/md0

  • To confirm that the data array is mounted and NVSM is running, run:

    findmnt /raid
systemctl is-active nvsm

  • Run the systemctl command below to check if cachfilesd is enabled. The expected result is enabled for RAID 0 and disabled for RAID 5:

    systemctl is-enabled cachefilesd

Running NGC Containers#

This section provides information about how to run NGC containers with your DGX system.

Obtaining an NGC Account#

Here is some information about how you can obtain an NGC account.

NVIDIA NGC provides simple access to GPU-optimized software for deep learning, machine learning, and high-performance computing (HPC). An NGC account grants you access to these tools and gives you the ability to set up a private registry to manage your customized software.

If you are the organization administrator for your DGX system purchase, work with NVIDIA Enterprise Support to set up an NGC enterprise account. Refer to the NGC Private Registry User Guide for more information about getting an NGC enterprise account.

Running NGC Containers with GPU Support#

To run GPU-enabled containers with Docker, use the nvidia-container-toolkit package. The toolkit provides the NVIDIA Container Runtime integration that makes the host GPU devices and driver libraries available to containers. Do not install the legacy nvidia-docker2 package.

  1. Confirm that the NVIDIA driver is working on the host.

    nvidia-smi

  2. Install the NVIDIA Container Toolkit if it is not already installed.

    sudo apt update
sudo apt install -y nvidia-container-toolkit

  3. Configure Docker to use the NVIDIA Container Runtime, then restart Docker.

    sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

  4. Verify GPU access before pulling a framework container.

    Use a published nvidia/cuda image tag whose CUDA runtime version is supported by the installed NVIDIA driver. The CUDA version shown by nvidia-smi is the maximum CUDA version supported by that driver.

    docker run --rm --gpus all nvidia/cuda:<cuda-version>-base-<ubuntu-version> nvidia-smi

    Select an actual tag from the NVIDIA CUDA container image tags. For example:

    docker run --rm --gpus all nvidia/cuda:13.2.0-base-ubuntu24.04 nvidia-smi

    The key rule is driver-to-CUDA-runtime compatibility; containers can use a different Ubuntu release from the host. NVIDIA documents backward compatibility with older CUDA runtimes on newer drivers, while newer runtimes on older drivers need explicitly supported compatibility arrangements. See CUDA compatibility guidance for more information.

  5. To run an NGC container with all GPUs, follow instructions in the NGC Catalog User Guide.

For more information, refer to the NVIDIA Container Toolkit installation guide and the NGC Catalog User Guide.