Troubleshooting#
This page documents solutions to common issues that you might encounter.
Hugepages Issues#
Normally the hugepages settings are updated through the /etc/default/grub configuration file. However, depending on the version of operating system, the settings changes
may become overwritten by another configuration file: /etc/grub.
Remove Old CUDA Toolkit and Driver#
If the system has an old version installed, run the following to remove the CUDA Toolkit and driver :
sudo apt-get --purge remove "*cublas*" "*cufft*" "*curand*" "*cusolver*" "*cusparse*" "*npp*" "*nvjpeg*" "cuda*" "nsight*" "*nvidia*"
sudo apt-get autoremove
How to Fix Apt Update Error Due to Incorrect System Time#
You may see the apt update error if the system time is incorrect.
E: Release file for https://download.docker.com/linux/ubuntu/dists/focal/InRelease is not valid yet (invalid for another 2d 10h 51min 11s).
Updates for this repository will not be applied.
Run the following commands to set the date and time via NTP once (this will not enable the NTP service):
sudo apt-get install ntpdate
sudo ntpdate -s pool.ntp.org
How to Resize the Default LVM Volume#
When installing Ubuntu 22.04 server, it partitions the whole disk but only creates a 200GB logical volume. This is what you will see on a newly installed devkit:
# Devkit has 1TB SSD but default lv uses only 200GB
lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
loop0 7:0 0 55.5M 1 loop /snap/core18/2246
loop1 7:1 0 55.5M 1 loop /snap/core18/2253
loop2 7:2 0 67.3M 1 loop /snap/lxd/21545
loop3 7:3 0 67.2M 1 loop /snap/lxd/21835
loop4 7:4 0 61.9M 1 loop /snap/core20/1242
loop5 7:5 0 61.9M 1 loop /snap/core20/1169
loop6 7:6 0 32.5M 1 loop /snap/snapd/13640
loop7 7:7 0 42.2M 1 loop /snap/snapd/14066
sda 8:0 0 894.3G 0 disk
├─sda1 8:1 0 512M 0 part /boot/efi
├─sda2 8:2 0 1G 0 part /boot
└─sda3 8:3 0 892.8G 0 part
└─ubuntu--vg-ubuntu--lv 253:0 0 200G 0 lvm /
The following commands resize the logic volume to use the entire disk, then resize the file system to use the entire logic volume.
# Test mode first
sudo lvresize -t -v -l +100%FREE /dev/mapper/ubuntu--vg-ubuntu--lv
# Remove -t if test mode succeeds
sudo lvresize -v -l +100%FREE /dev/mapper/ubuntu--vg-ubuntu--lv
lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
loop0 7:0 0 55.5M 1 loop /snap/core18/2246
loop1 7:1 0 55.5M 1 loop /snap/core18/2253
loop2 7:2 0 67.3M 1 loop /snap/lxd/21545
loop3 7:3 0 67.2M 1 loop /snap/lxd/21835
loop4 7:4 0 61.9M 1 loop /snap/core20/1242
loop5 7:5 0 61.9M 1 loop /snap/core20/1169
loop6 7:6 0 32.5M 1 loop /snap/snapd/13640
loop7 7:7 0 42.2M 1 loop /snap/snapd/14066
sda 8:0 0 894.3G 0 disk
├─sda1 8:1 0 512M 0 part /boot/efi
├─sda2 8:2 0 1G 0 part /boot
└─sda3 8:3 0 892.8G 0 part
└─ubuntu--vg-ubuntu--lv 253:0 0 892.8G 0 lvm /
# Resize file system
sudo resize2fs -p /dev/mapper/ubuntu--vg-ubuntu--lv
df -h -T
Filesystem Type Size Used Avail Use% Mounted on
udev devtmpfs 39G 0 39G 0% /dev
tmpfs tmpfs 9.4G 2.0M 9.4G 1% /run
/dev/mapper/ubuntu--vg-ubuntu--lv ext4 878G 77G 764G 10% /
tmpfs tmpfs 47G 0 47G 0% /dev/shm
tmpfs tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs tmpfs 47G 0 47G 0% /sys/fs/cgroup
/dev/sda2 ext4 976M 460M 450M 51% /boot
/dev/loop0 squashfs 56M 56M 0 100% /snap/core18/2246
/dev/sda1 vfat 511M 5.3M 506M 2% /boot/efi
/dev/loop1 squashfs 56M 56M 0 100% /snap/core18/2253
/dev/loop5 squashfs 62M 62M 0 100% /snap/core20/1169
/dev/loop2 squashfs 68M 68M 0 100% /snap/lxd/21545
/dev/loop4 squashfs 62M 62M 0 100% /snap/core20/1242
/dev/loop6 squashfs 33M 33M 0 100% /snap/snapd/13640
/dev/loop3 squashfs 68M 68M 0 100% /snap/lxd/21835
/dev/loop7 squashfs 43M 43M 0 100% /snap/snapd/14066
overlay overlay 878G 77G 764G 10% /var/lib/docker/overlay2/851cbfd83b022a24f61fb0f87a007c56da8065a7528f6b661bf45d3d65ccc787/merged
tmpfs tmpfs 9.4G 4.0K 9.4G 1% / run/user/1000
How to Identify the NIC Interface Name and MAC Address#
Use the sudo lshw -c network |grep -i 'product\|bus info\|name\|serial command to find the bus address and MAC address of each NIC on the system. Here is an example:
sudo lshw -c network |grep -i 'product\|bus info\|name\|serial'
product: I210 Gigabit Network Connection
bus info: pci@0000:05:00.0
logical name: eno1
serial: 18:c0:4d:79:49:b6
product: I210 Gigabit Network Connection
bus info: pci@0000:06:00.0
logical name: enp6s0
serial: 18:c0:4d:79:49:b7
product: MT2892 Family [ConnectX-6 Dx]
bus info: pci@0000:b5:00.0
logical name: ens6f0
serial: b8:ce:f6:33:fd:ee
product: MT2892 Family [ConnectX-6 Dx]
bus info: pci@0000:b5:00.1
logical name: ens6f1
serial: b8:ce:f6:33:fd:ef
How to Upgrade Host OS from Ubuntu 22.04 to 24.04#
Phase 1: Pre-Upgrade#
# ── Hold kernel packages to avoid unnecessary DKMS rebuilds ──
sudo apt-mark hold linux-headers-* linux-image-* linux-modules-* \
linux-generic-hwe-* linux-headers-generic-hwe-* linux-image-generic-hwe-*
# ── Fix any broken state from prior run ──
sudo apt --fix-broken install -y
sudo dpkg --configure -a
# ── Update ONLY userspace packages (no kernel, no DKMS rebuilds) ──
sudo apt update && sudo apt upgrade -y
# ── Unhold before do-release-upgrade (it handles kernel transition) ──
sudo apt-mark unhold linux-headers-* linux-image-* linux-modules-* \
linux-generic-hwe-* linux-headers-generic-hwe-* linux-image-generic-hwe-*
# ── Clean up ──
sudo apt autoremove -y && sudo apt autoclean
Phase 2: Verify Starting State#
lsb_release -a # Should show 22.04
uname -r # Current kernel
nvidia-smi # Verify GPU visible
ofed_info -s # Note OFED version
dkms status # List all DKMS modules — save for reference
Phase 3: Prepare for Upgrade#
# ── Disable third-party repos ──
sudo mv /etc/apt/sources.list.d /etc/apt/sources.list.d.bak
sudo mkdir /etc/apt/sources.list.d
# ── Record current state for reference ──
dkms status | tee ~/dkms_modules_22.04.txt
# ── Uninstall NVIDIA driver ──
sudo dkms remove nvidia/590.48.01 --all 2>/dev/null || true
sudo apt remove --purge -y 'nvidia-driver-*' 'cuda-*' 2>/dev/null || true
# ── Uninstall DOCA/MOFED ──
for f in $(dpkg --list | grep -E 'doca|flexio|dpa-gdbserver|dpa-stats|dpaeumgmt' | awk '{print $2}'); do
sudo apt remove --purge $f -y
done
sudo /usr/sbin/ofed_uninstall.sh --force 2>/dev/null || true
# ── Remove ALL remaining DKMS modules ──
sudo dkms remove iser/25.10.OFED.25.10.1.7.1.1 --all 2>/dev/null || true
sudo dkms remove isert/25.10.OFED.25.10.1.7.1.1 --all 2>/dev/null || true
sudo dkms remove gdrdrv/2.5 --all 2>/dev/null || true
sudo dkms remove knem/1.1.4.90mlnx4 --all 2>/dev/null || true
sudo dkms remove mlnx-ofed-kernel/25.10.OFED.25.10.1.7.1.1 --all 2>/dev/null || true
sudo dkms remove srp/25.10.OFED.25.10.1.7.1.1 --all 2>/dev/null || true
sudo dkms remove xpmem/2510.0.16 --all 2>/dev/null || true
sudo dkms remove kernel-mft-dkms/4.34.1.10 --all 2>/dev/null || true
# ── VERIFY: dkms status MUST be empty ──
dkms status
sudo apt-get autoremove -y
# ── Ensure upgrade tool is ready ──
sudo apt install update-manager-core -y
sudo sed -i 's/Prompt=.*/Prompt=lts/' /etc/update-manager/release-upgrades
Note
Critical checkpoint: dkms status must return empty before proceeding.
Phase 4a: Pre-Upgrade Preparation#
Quick check: Run the commands below to see if this phase can be skipped.
sudo apt update
apt list --upgradable
apt-mark showhold
cat /var/run/reboot-required 2>/dev/null || echo "No reboot required"
✅ Skip to Phase 4b if ALL three are true:
No upgradable packages listed
No held packages listed
No reboot required
⚠️ Otherwise, run the full Phase 4a:
# ── Unhold all held packages automatically ──
sudo apt-mark showhold | xargs -r sudo apt-mark unhold
# ── Fix any broken/half-configured packages ──
sudo dpkg --configure -a
sudo apt --fix-broken install -y
# ── Install ALL pending updates (including kernel transitions) ──
sudo apt update
sudo apt dist-upgrade -y
# ── Verify nothing is left pending ──
apt list --upgradable
# Must return: "Listing... Done" with NO packages listed
# ── Clean up ──
sudo apt autoremove -y
sudo apt autoclean
# ── Reboot (required if kernel packages changed) ──
sudo reboot
Phase 4b: Perform the Upgrade#
# ── Start a persistent session (survives SSH drops) ──
tmux new -s upgrade
# ── Perform the upgrade ──
sudo do-release-upgrade
During upgrade prompts:
Modified config files → keep local version
Remove obsolete packages → review, say yes
Reboot at end → yes
If SSH drops, reconnect on port 1022.
Phase 5: Post-Upgrade Verification#
lsb_release -a # Should show 24.04 Noble Numbat
uname -r # Temporary generic 24.04 kernel
cat /etc/os-release
# ── Refresh package index only ──
sudo apt update
# ── Remove leftover 22.04 packages ──
sudo apt autoremove -y