Is this page helpful?

Appendix#

ATS and ACS configuration#

PCIe Address Translation Services (ATS) and Access Control Services (ACS) control how translated peer transactions traverse the IOMMU and PCIe hierarchy. The tested configuration enables ATS on every ConnectX-8 adapter and sets ACS on every ACS-capable bridge on the paths to the passed-through GPUs and ConnectX-8 functions.

ATS must be enabled on the NIC, and ACS on the PCIe switches and root ports must be configured to allow ATS.

Apply ACS only to the bridges on the paths to the passed-through GPUs and ConnectX-8 functions. Do not write one ACS value to every ACS-capable device in the system.

Enabling ATS on NVIDIA NICs#

Enable ATS on every ConnectX-8 function with mlxconfig, reboot the host, and confirm that every adapter reports True(1). The setting is stored in the adapter firmware and persists across reboots.

Here is the expected output:

=== mlxconfig ATS ===
----- 17:00.0 -----
Device type: ConnectX8
Name: P6612_Ax
ATS_ENABLED True(1)
----- 39:00.0 -----
Device type: ConnectX8
Name: P6612_Ax
ATS_ENABLED True(1)
----- 5f:00.0 -----
Device type: ConnectX8
Name: P6612_Ax
ATS_ENABLED True(1)
----- 70:00.0 -----
Device type: ConnectX8
Name: P6612_Ax
ATS_ENABLED True(1)
----- 97:00.0 -----
Device type: ConnectX8
Name: P6612_Ax
ATS_ENABLED True(1)
----- b9:00.0 -----
Device type: ConnectX8
Name: P6612_Ax
ATS_ENABLED True(1)
----- dc:00.0 -----
Device type: ConnectX8
Name: P6612_Ax
ATS_ENABLED True(1)
----- ed:00.0 -----
Device type: ConnectX8
Name: P6612_Ax
ATS_ENABLED True(1)

Enable ATS on all sixteen ConnectX-8 functions, then reboot the host:

for bdf in $(lspci -D -d 15b3:1023 | cut -d' ' -f1); do
sudo mlxconfig -d "$bdf" -y set ATS_ENABLED=1
done
# After the reboot, verify; expect True(1) on all 16 functions
for bdf in $(lspci -D -d 15b3:1023 | cut -d' ' -f1); do
echo "$bdf $(sudo mlxconfig -d "$bdf" q ATS_ENABLED | awk '$1=="ATS_ENABLED"{print $2}')"
done

Configuring ACS#

ACS capability and control bits are exposed on PCIe root ports and switch ports. For the relevant path, inspect Source Validation, Translation Blocking, P2P Request and Completion Redirection, Upstream Forwarding, Egress Control and Direct Translated P2P.

# Read-only capability audit
sudo lspci -Dvv | grep -B3 -A3 -E 'Access Control Services|ACSCap:|ACSCtl:'

# Inspect one exact bridge on the GPU/NIC path
sudo lspci -s <bridge-bdf> -vv | grep -A2 -E 'Access Control Services|ACSCap:|ACSCtl:'

Example from the upstream B300 path (0000:16:00.0), before and after the VM-mode setting:

----- 0000:16:00.0 -----
Capabilities: [160 v1] Access Control Services
ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl+ DirectTrans+
ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
ACSCtl (VM mode): SrcValid+ TransBlk- ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans+

Please review the following when setting ACS:

  1. Set ACS only on the ACS-capable bridges (root ports and PCIe switch ports) on the paths to the GPUs and ConnectX-8 functions.

  2. Write 0x5d (Source Validation, P2P Request Redirect, P2P Completion Redirect, Upstream Forwarding and Direct Translated P2P) on bridges that support Direct Translated P2P, and 0x1d (the same without Direct Translated P2P) on the others.

  3. On the tested system the script below sets 24 bridges to 0x5d and 8 to 0x1d; the remaining 16 bridges on those paths have no ACS capability.

  4. The DGX OS nvidia-acs-disable service clears ACS at every host boot, so run the script after each boot, with the VM shut off.

# Run on the host after every boot, with the VM shut off
for d in $(lspci -D -d 10de:3182 | cut -d' ' -f1) $(lspci -D -d 15b3:1023 | cut -d' ' -f1); do
readlink -f /sys/bus/pci/devices/$d | tr '/' '\n' | grep -E '^[0-9a-f]{4}:[0-9a-f]{2}:' | head -n -1
done | sort -u | while read -r b; do
sudo lspci -s "$b" -vv | grep -q 'Access Control Services' || continue
if sudo lspci -s "$b" -vv | grep -q 'ACSCap:.*DirectTrans+'; then v=005d; else v=001d; fi
sudo setpci -s "$b" ECAP_ACS+6.w=$v:007f
echo "$b ACSCtl=$(sudo setpci -s "$b" ECAP_ACS+6.w)"
done

Example physical topology map for B300 NVL8 8-GPU platform#

The diagram and raw output in this section were generated from the live B300 NVL8 host while the VM was shut off, all eight GPUs were bound to the NVIDIA driver, Fabric Manager was active, and the host reported a healthy NV18 fabric.

B300 NVL8 Physical Topology#

Figure 7. Physical topology map for the tested B300 NVL8.

Figure 7. Physical topology map for the tested B300 NVL8.

GPU

Host BDF

NUMA

Paired ConnectX-8 PFs

0

1a:00.0

0

17:00.0, 17:00.1

1

3c:00.0

0

39:00.0, 39:00.1

2

62:00.0

0

5f:00.0, 5f:00.1

3

73:00.0

0

70:00.0, 70:00.1

4

9a:00.0

1

97:00.0, 97:00.1

5

bc:00.0

1

b9:00.0, b9:00.1

6

df:00.0

1

dc:00.0, dc:00.1

7

f0:00.0

1

ed:00.0, ed:00.1

Full output from lstopo-no-graphics --of console follows. It is reproduced as captured so readers can see CPU caches, processing units, bridges, endpoints, network objects and block devices. Device addresses should be rediscovered on every target system.

Machine (2016GB total)
Package L#0
NUMANode L#0 (P#0 1008GB)
L3 L#0 (336MB)
L2 L#0 (2048KB) + L1d L#0 (48KB) + L1i L#0 (64KB) + Core L#0
PU L#0 (P#0)
PU L#1 (P#128)
L2 L#1 (2048KB) + L1d L#1 (48KB) + L1i L#1 (64KB) + Core L#1
PU L#2 (P#1)
PU L#3 (P#129)
L2 L#2 (2048KB) + L1d L#2 (48KB) + L1i L#2 (64KB) + Core L#2
PU L#4 (P#2)
PU L#5 (P#130)
L2 L#3 (2048KB) + L1d L#3 (48KB) + L1i L#3 (64KB) + Core L#3
PU L#6 (P#3)
PU L#7 (P#131)
L2 L#4 (2048KB) + L1d L#4 (48KB) + L1i L#4 (64KB) + Core L#4
PU L#8 (P#4)
PU L#9 (P#132)
L2 L#5 (2048KB) + L1d L#5 (48KB) + L1i L#5 (64KB) + Core L#5
PU L#10 (P#5)
PU L#11 (P#133)
L2 L#6 (2048KB) + L1d L#6 (48KB) + L1i L#6 (64KB) + Core L#6
PU L#12 (P#6)
PU L#13 (P#134)
L2 L#7 (2048KB) + L1d L#7 (48KB) + L1i L#7 (64KB) + Core L#7
PU L#14 (P#7)
PU L#15 (P#135)
L2 L#8 (2048KB) + L1d L#8 (48KB) + L1i L#8 (64KB) + Core L#8
PU L#16 (P#8)
PU L#17 (P#136)
L2 L#9 (2048KB) + L1d L#9 (48KB) + L1i L#9 (64KB) + Core L#9
PU L#18 (P#9)
PU L#19 (P#137)
L2 L#10 (2048KB) + L1d L#10 (48KB) + L1i L#10 (64KB) + Core L#10
PU L#20 (P#10)
PU L#21 (P#138)
L2 L#11 (2048KB) + L1d L#11 (48KB) + L1i L#11 (64KB) + Core L#11
PU L#22 (P#11)
PU L#23 (P#139)
L2 L#12 (2048KB) + L1d L#12 (48KB) + L1i L#12 (64KB) + Core L#12
PU L#24 (P#12)
PU L#25 (P#140)
L2 L#13 (2048KB) + L1d L#13 (48KB) + L1i L#13 (64KB) + Core L#13
PU L#26 (P#13)
PU L#27 (P#141)
L2 L#14 (2048KB) + L1d L#14 (48KB) + L1i L#14 (64KB) + Core L#14
PU L#28 (P#14)
PU L#29 (P#142)
L2 L#15 (2048KB) + L1d L#15 (48KB) + L1i L#15 (64KB) + Core L#15
PU L#30 (P#15)
PU L#31 (P#143)
L2 L#16 (2048KB) + L1d L#16 (48KB) + L1i L#16 (64KB) + Core L#16
PU L#32 (P#16)
PU L#33 (P#144)
L2 L#17 (2048KB) + L1d L#17 (48KB) + L1i L#17 (64KB) + Core L#17
PU L#34 (P#17)
PU L#35 (P#145)
L2 L#18 (2048KB) + L1d L#18 (48KB) + L1i L#18 (64KB) + Core L#18
PU L#36 (P#18)
PU L#37 (P#146)
L2 L#19 (2048KB) + L1d L#19 (48KB) + L1i L#19 (64KB) + Core L#19
PU L#38 (P#19)
PU L#39 (P#147)
L2 L#20 (2048KB) + L1d L#20 (48KB) + L1i L#20 (64KB) + Core L#20
PU L#40 (P#20)
PU L#41 (P#148)
L2 L#21 (2048KB) + L1d L#21 (48KB) + L1i L#21 (64KB) + Core L#21
PU L#42 (P#21)
PU L#43 (P#149)
L2 L#22 (2048KB) + L1d L#22 (48KB) + L1i L#22 (64KB) + Core L#22
PU L#44 (P#22)
PU L#45 (P#150)
L2 L#23 (2048KB) + L1d L#23 (48KB) + L1i L#23 (64KB) + Core L#23
PU L#46 (P#23)
PU L#47 (P#151)
L2 L#24 (2048KB) + L1d L#24 (48KB) + L1i L#24 (64KB) + Core L#24
PU L#48 (P#24)
PU L#49 (P#152)
L2 L#25 (2048KB) + L1d L#25 (48KB) + L1i L#25 (64KB) + Core L#25
PU L#50 (P#25)
PU L#51 (P#153)
L2 L#26 (2048KB) + L1d L#26 (48KB) + L1i L#26 (64KB) + Core L#26
PU L#52 (P#26)
PU L#53 (P#154)
L2 L#27 (2048KB) + L1d L#27 (48KB) + L1i L#27 (64KB) + Core L#27
PU L#54 (P#27)
PU L#55 (P#155)
L2 L#28 (2048KB) + L1d L#28 (48KB) + L1i L#28 (64KB) + Core L#28
PU L#56 (P#28)
PU L#57 (P#156)
L2 L#29 (2048KB) + L1d L#29 (48KB) + L1i L#29 (64KB) + Core L#29
PU L#58 (P#29)
PU L#59 (P#157)
L2 L#30 (2048KB) + L1d L#30 (48KB) + L1i L#30 (64KB) + Core L#30
PU L#60 (P#30)
PU L#61 (P#158)
L2 L#31 (2048KB) + L1d L#31 (48KB) + L1i L#31 (64KB) + Core L#31
PU L#62 (P#31)
PU L#63 (P#159)
L2 L#32 (2048KB) + L1d L#32 (48KB) + L1i L#32 (64KB) + Core L#32
PU L#64 (P#32)
PU L#65 (P#160)
L2 L#33 (2048KB) + L1d L#33 (48KB) + L1i L#33 (64KB) + Core L#33
PU L#66 (P#33)
PU L#67 (P#161)
L2 L#34 (2048KB) + L1d L#34 (48KB) + L1i L#34 (64KB) + Core L#34
PU L#68 (P#34)
PU L#69 (P#162)
L2 L#35 (2048KB) + L1d L#35 (48KB) + L1i L#35 (64KB) + Core L#35
PU L#70 (P#35)
PU L#71 (P#163)
L2 L#36 (2048KB) + L1d L#36 (48KB) + L1i L#36 (64KB) + Core L#36
PU L#72 (P#36)
PU L#73 (P#164)
L2 L#37 (2048KB) + L1d L#37 (48KB) + L1i L#37 (64KB) + Core L#37
PU L#74 (P#37)
PU L#75 (P#165)
L2 L#38 (2048KB) + L1d L#38 (48KB) + L1i L#38 (64KB) + Core L#38
PU L#76 (P#38)
PU L#77 (P#166)
L2 L#39 (2048KB) + L1d L#39 (48KB) + L1i L#39 (64KB) + Core L#39
PU L#78 (P#39)
PU L#79 (P#167)
L2 L#40 (2048KB) + L1d L#40 (48KB) + L1i L#40 (64KB) + Core L#40
PU L#80 (P#40)
PU L#81 (P#168)
L2 L#41 (2048KB) + L1d L#41 (48KB) + L1i L#41 (64KB) + Core L#41
PU L#82 (P#41)
PU L#83 (P#169)
L2 L#42 (2048KB) + L1d L#42 (48KB) + L1i L#42 (64KB) + Core L#42
PU L#84 (P#42)
PU L#85 (P#170)
L2 L#43 (2048KB) + L1d L#43 (48KB) + L1i L#43 (64KB) + Core L#43
PU L#86 (P#43)
PU L#87 (P#171)
L2 L#44 (2048KB) + L1d L#44 (48KB) + L1i L#44 (64KB) + Core L#44
PU L#88 (P#44)
PU L#89 (P#172)
L2 L#45 (2048KB) + L1d L#45 (48KB) + L1i L#45 (64KB) + Core L#45
PU L#90 (P#45)
PU L#91 (P#173)
L2 L#46 (2048KB) + L1d L#46 (48KB) + L1i L#46 (64KB) + Core L#46
PU L#92 (P#46)
PU L#93 (P#174)
L2 L#47 (2048KB) + L1d L#47 (48KB) + L1i L#47 (64KB) + Core L#47
PU L#94 (P#47)
PU L#95 (P#175)
L2 L#48 (2048KB) + L1d L#48 (48KB) + L1i L#48 (64KB) + Core L#48
PU L#96 (P#48)
PU L#97 (P#176)
L2 L#49 (2048KB) + L1d L#49 (48KB) + L1i L#49 (64KB) + Core L#49
PU L#98 (P#49)
PU L#99 (P#177)
L2 L#50 (2048KB) + L1d L#50 (48KB) + L1i L#50 (64KB) + Core L#50
PU L#100 (P#50)
PU L#101 (P#178)
L2 L#51 (2048KB) + L1d L#51 (48KB) + L1i L#51 (64KB) + Core L#51
PU L#102 (P#51)
PU L#103 (P#179)
L2 L#52 (2048KB) + L1d L#52 (48KB) + L1i L#52 (64KB) + Core L#52
PU L#104 (P#52)
PU L#105 (P#180)
L2 L#53 (2048KB) + L1d L#53 (48KB) + L1i L#53 (64KB) + Core L#53
PU L#106 (P#53)
PU L#107 (P#181)
L2 L#54 (2048KB) + L1d L#54 (48KB) + L1i L#54 (64KB) + Core L#54
PU L#108 (P#54)
PU L#109 (P#182)
L2 L#55 (2048KB) + L1d L#55 (48KB) + L1i L#55 (64KB) + Core L#55
PU L#110 (P#55)
PU L#111 (P#183)
L2 L#56 (2048KB) + L1d L#56 (48KB) + L1i L#56 (64KB) + Core L#56
PU L#112 (P#56)
PU L#113 (P#184)
L2 L#57 (2048KB) + L1d L#57 (48KB) + L1i L#57 (64KB) + Core L#57
PU L#114 (P#57)
PU L#115 (P#185)
L2 L#58 (2048KB) + L1d L#58 (48KB) + L1i L#58 (64KB) + Core L#58
PU L#116 (P#58)
PU L#117 (P#186)
L2 L#59 (2048KB) + L1d L#59 (48KB) + L1i L#59 (64KB) + Core L#59
PU L#118 (P#59)
PU L#119 (P#187)
L2 L#60 (2048KB) + L1d L#60 (48KB) + L1i L#60 (64KB) + Core L#60
PU L#120 (P#60)
PU L#121 (P#188)
L2 L#61 (2048KB) + L1d L#61 (48KB) + L1i L#61 (64KB) + Core L#61
PU L#122 (P#61)
PU L#123 (P#189)
L2 L#62 (2048KB) + L1d L#62 (48KB) + L1i L#62 (64KB) + Core L#62
PU L#124 (P#62)
PU L#125 (P#190)
L2 L#63 (2048KB) + L1d L#63 (48KB) + L1i L#63 (64KB) + Core L#63
PU L#126 (P#63)
PU L#127 (P#191)
HostBridge
PCI 01:00.0 (Co-Processor)
HostBridge
PCI 03:00.0 (Co-Processor)
HostBridge
PCI 0b:00.0 (Co-Processor)
HostBridge
PCI 0d:00.0 (Co-Processor)
HostBridge
PCIBridge
PCIBridge
PCIBridge
PCI 17:00.0 (Ethernet)
Net "eno10np0"
OpenFabrics "mlx5_4"
PCI 17:00.1 (Ethernet)
Net "enp23s0f1np1"
OpenFabrics "mlx5_5"
PCIBridge
PCIBridge
PCIBridge
PCI 1a:00.0 (3D)
CoProc(OpenCL) "opencl0d0"
HostBridge
PCIBridge
PCI 27:00.0 (InfiniBand)
Net "ibo4"
OpenFabrics "mlx5_0"
PCI 27:00.1 (InfiniBand)
Net "ibs301f1"
OpenFabrics "mlx5_1"
PCI 27:00.2 (InfiniBand)
Net "ibs301f2"
OpenFabrics "mlx5_2"
PCI 27:00.3 (InfiniBand)
Net "ibs301f3"
OpenFabrics "mlx5_3"
PCIBridge
PCIBridge
PCI 29:00.0 (VGA)
PCIBridge
PCI 2a:00.0 (Ethernet)
Net "eno1"
HostBridge
PCIBridge
PCIBridge
PCIBridge
PCI 39:00.0 (Ethernet)
Net "eno12np0"
OpenFabrics "mlx5_6"
PCI 39:00.1 (Ethernet)
Net "enp57s0f1np1"
OpenFabrics "mlx5_7"
PCIBridge
PCIBridge
PCIBridge
PCI 3c:00.0 (3D)
CoProc(OpenCL) "opencl0d1"
HostBridge
PCIBridge
PCIBridge
PCIBridge
PCIBridge
PCIBridge
PCI 4d:00.0 (NVMExp)
Block(Disk) "nvme1n1"
PCIBridge
PCI 4e:00.0 (NVMExp)
Block(Disk) "nvme2n1"
PCIBridge
PCI 4f:00.0 (NVMExp)
Block(Disk) "nvme5n1"
PCIBridge
PCI 50:00.0 (NVMExp)
Block(Disk) "nvme7n1"
PCIBridge
PCIBridge
PCIBridge
PCI 53:00.0 (Ethernet)
Net "enp83s0f0np0"
OpenFabrics "mlx5_8"
PCI 53:00.1 (Ethernet)
Net "enp83s0f1np1"
OpenFabrics "mlx5_9"
HostBridge
PCIBridge
PCIBridge
PCIBridge
PCI 5f:00.0 (Ethernet)
Net "eno13np0"
OpenFabrics "mlx5_10"
PCI 5f:00.1 (Ethernet)
Net "enp95s0f1np1"
OpenFabrics "mlx5_11"
PCIBridge
PCIBridge
PCIBridge
PCI 62:00.0 (3D)
CoProc(OpenCL) "opencl0d2"
HostBridge
PCIBridge
PCIBridge
PCIBridge
PCI 70:00.0 (Ethernet)
Net "eno11np0"
OpenFabrics "mlx5_12"
PCI 70:00.1 (Ethernet)
Net "enp112s0f1np1"
OpenFabrics "mlx5_13"
PCIBridge
PCIBridge
PCIBridge
PCI 73:00.0 (3D)
CoProc(OpenCL) "opencl0d3"
Package L#1
NUMANode L#1 (P#1 1008GB)
L3 L#1 (336MB)
L2 L#64 (2048KB) + L1d L#64 (48KB) + L1i L#64 (64KB) + Core L#64
PU L#128 (P#64)
PU L#129 (P#192)
L2 L#65 (2048KB) + L1d L#65 (48KB) + L1i L#65 (64KB) + Core L#65
PU L#130 (P#65)
PU L#131 (P#193)
L2 L#66 (2048KB) + L1d L#66 (48KB) + L1i L#66 (64KB) + Core L#66
PU L#132 (P#66)
PU L#133 (P#194)
L2 L#67 (2048KB) + L1d L#67 (48KB) + L1i L#67 (64KB) + Core L#67
PU L#134 (P#67)
PU L#135 (P#195)
L2 L#68 (2048KB) + L1d L#68 (48KB) + L1i L#68 (64KB) + Core L#68
PU L#136 (P#68)
PU L#137 (P#196)
L2 L#69 (2048KB) + L1d L#69 (48KB) + L1i L#69 (64KB) + Core L#69
PU L#138 (P#69)
PU L#139 (P#197)
L2 L#70 (2048KB) + L1d L#70 (48KB) + L1i L#70 (64KB) + Core L#70
PU L#140 (P#70)
PU L#141 (P#198)
L2 L#71 (2048KB) + L1d L#71 (48KB) + L1i L#71 (64KB) + Core L#71
PU L#142 (P#71)
PU L#143 (P#199)
L2 L#72 (2048KB) + L1d L#72 (48KB) + L1i L#72 (64KB) + Core L#72
PU L#144 (P#72)
PU L#145 (P#200)
L2 L#73 (2048KB) + L1d L#73 (48KB) + L1i L#73 (64KB) + Core L#73
PU L#146 (P#73)
PU L#147 (P#201)
L2 L#74 (2048KB) + L1d L#74 (48KB) + L1i L#74 (64KB) + Core L#74
PU L#148 (P#74)
PU L#149 (P#202)
L2 L#75 (2048KB) + L1d L#75 (48KB) + L1i L#75 (64KB) + Core L#75
PU L#150 (P#75)
PU L#151 (P#203)
L2 L#76 (2048KB) + L1d L#76 (48KB) + L1i L#76 (64KB) + Core L#76
PU L#152 (P#76)
PU L#153 (P#204)
L2 L#77 (2048KB) + L1d L#77 (48KB) + L1i L#77 (64KB) + Core L#77
PU L#154 (P#77)
PU L#155 (P#205)
L2 L#78 (2048KB) + L1d L#78 (48KB) + L1i L#78 (64KB) + Core L#78
PU L#156 (P#78)
PU L#157 (P#206)
L2 L#79 (2048KB) + L1d L#79 (48KB) + L1i L#79 (64KB) + Core L#79
PU L#158 (P#79)
PU L#159 (P#207)
L2 L#80 (2048KB) + L1d L#80 (48KB) + L1i L#80 (64KB) + Core L#80
PU L#160 (P#80)
PU L#161 (P#208)
L2 L#81 (2048KB) + L1d L#81 (48KB) + L1i L#81 (64KB) + Core L#81
PU L#162 (P#81)
PU L#163 (P#209)
L2 L#82 (2048KB) + L1d L#82 (48KB) + L1i L#82 (64KB) + Core L#82
PU L#164 (P#82)
PU L#165 (P#210)
L2 L#83 (2048KB) + L1d L#83 (48KB) + L1i L#83 (64KB) + Core L#83
PU L#166 (P#83)
PU L#167 (P#211)
L2 L#84 (2048KB) + L1d L#84 (48KB) + L1i L#84 (64KB) + Core L#84
PU L#168 (P#84)
PU L#169 (P#212)
L2 L#85 (2048KB) + L1d L#85 (48KB) + L1i L#85 (64KB) + Core L#85
PU L#170 (P#85)
PU L#171 (P#213)
L2 L#86 (2048KB) + L1d L#86 (48KB) + L1i L#86 (64KB) + Core L#86
PU L#172 (P#86)
PU L#173 (P#214)
L2 L#87 (2048KB) + L1d L#87 (48KB) + L1i L#87 (64KB) + Core L#87
PU L#174 (P#87)
PU L#175 (P#215)
L2 L#88 (2048KB) + L1d L#88 (48KB) + L1i L#88 (64KB) + Core L#88
PU L#176 (P#88)
PU L#177 (P#216)
L2 L#89 (2048KB) + L1d L#89 (48KB) + L1i L#89 (64KB) + Core L#89
PU L#178 (P#89)
PU L#179 (P#217)
L2 L#90 (2048KB) + L1d L#90 (48KB) + L1i L#90 (64KB) + Core L#90
PU L#180 (P#90)
PU L#181 (P#218)
L2 L#91 (2048KB) + L1d L#91 (48KB) + L1i L#91 (64KB) + Core L#91
PU L#182 (P#91)
PU L#183 (P#219)
L2 L#92 (2048KB) + L1d L#92 (48KB) + L1i L#92 (64KB) + Core L#92
PU L#184 (P#92)
PU L#185 (P#220)
L2 L#93 (2048KB) + L1d L#93 (48KB) + L1i L#93 (64KB) + Core L#93
PU L#186 (P#93)
PU L#187 (P#221)
L2 L#94 (2048KB) + L1d L#94 (48KB) + L1i L#94 (64KB) + Core L#94
PU L#188 (P#94)
PU L#189 (P#222)
L2 L#95 (2048KB) + L1d L#95 (48KB) + L1i L#95 (64KB) + Core L#95
PU L#190 (P#95)
PU L#191 (P#223)
L2 L#96 (2048KB) + L1d L#96 (48KB) + L1i L#96 (64KB) + Core L#96
PU L#192 (P#96)
PU L#193 (P#224)
L2 L#97 (2048KB) + L1d L#97 (48KB) + L1i L#97 (64KB) + Core L#97
PU L#194 (P#97)
PU L#195 (P#225)
L2 L#98 (2048KB) + L1d L#98 (48KB) + L1i L#98 (64KB) + Core L#98
PU L#196 (P#98)
PU L#197 (P#226)
L2 L#99 (2048KB) + L1d L#99 (48KB) + L1i L#99 (64KB) + Core L#99
PU L#198 (P#99)
PU L#199 (P#227)
L2 L#100 (2048KB) + L1d L#100 (48KB) + L1i L#100 (64KB) + Core L#100
PU L#200 (P#100)
PU L#201 (P#228)
L2 L#101 (2048KB) + L1d L#101 (48KB) + L1i L#101 (64KB) + Core L#101
PU L#202 (P#101)
PU L#203 (P#229)
L2 L#102 (2048KB) + L1d L#102 (48KB) + L1i L#102 (64KB) + Core L#102
PU L#204 (P#102)
PU L#205 (P#230)
L2 L#103 (2048KB) + L1d L#103 (48KB) + L1i L#103 (64KB) + Core L#103
PU L#206 (P#103)
PU L#207 (P#231)
L2 L#104 (2048KB) + L1d L#104 (48KB) + L1i L#104 (64KB) + Core L#104
PU L#208 (P#104)
PU L#209 (P#232)
L2 L#105 (2048KB) + L1d L#105 (48KB) + L1i L#105 (64KB) + Core L#105
PU L#210 (P#105)
PU L#211 (P#233)
L2 L#106 (2048KB) + L1d L#106 (48KB) + L1i L#106 (64KB) + Core L#106
PU L#212 (P#106)
PU L#213 (P#234)
L2 L#107 (2048KB) + L1d L#107 (48KB) + L1i L#107 (64KB) + Core L#107
PU L#214 (P#107)
PU L#215 (P#235)
L2 L#108 (2048KB) + L1d L#108 (48KB) + L1i L#108 (64KB) + Core L#108
PU L#216 (P#108)
PU L#217 (P#236)
L2 L#109 (2048KB) + L1d L#109 (48KB) + L1i L#109 (64KB) + Core L#109
PU L#218 (P#109)
PU L#219 (P#237)
L2 L#110 (2048KB) + L1d L#110 (48KB) + L1i L#110 (64KB) + Core L#110
PU L#220 (P#110)
PU L#221 (P#238)
L2 L#111 (2048KB) + L1d L#111 (48KB) + L1i L#111 (64KB) + Core L#111
PU L#222 (P#111)
PU L#223 (P#239)
L2 L#112 (2048KB) + L1d L#112 (48KB) + L1i L#112 (64KB) + Core L#112
PU L#224 (P#112)
PU L#225 (P#240)
L2 L#113 (2048KB) + L1d L#113 (48KB) + L1i L#113 (64KB) + Core L#113
PU L#226 (P#113)
PU L#227 (P#241)
L2 L#114 (2048KB) + L1d L#114 (48KB) + L1i L#114 (64KB) + Core L#114
PU L#228 (P#114)
PU L#229 (P#242)
L2 L#115 (2048KB) + L1d L#115 (48KB) + L1i L#115 (64KB) + Core L#115
PU L#230 (P#115)
PU L#231 (P#243)
L2 L#116 (2048KB) + L1d L#116 (48KB) + L1i L#116 (64KB) + Core L#116
PU L#232 (P#116)
PU L#233 (P#244)
L2 L#117 (2048KB) + L1d L#117 (48KB) + L1i L#117 (64KB) + Core L#117
PU L#234 (P#117)
PU L#235 (P#245)
L2 L#118 (2048KB) + L1d L#118 (48KB) + L1i L#118 (64KB) + Core L#118
PU L#236 (P#118)
PU L#237 (P#246)
L2 L#119 (2048KB) + L1d L#119 (48KB) + L1i L#119 (64KB) + Core L#119
PU L#238 (P#119)
PU L#239 (P#247)
L2 L#120 (2048KB) + L1d L#120 (48KB) + L1i L#120 (64KB) + Core L#120
PU L#240 (P#120)
PU L#241 (P#248)
L2 L#121 (2048KB) + L1d L#121 (48KB) + L1i L#121 (64KB) + Core L#121
PU L#242 (P#121)
PU L#243 (P#249)
L2 L#122 (2048KB) + L1d L#122 (48KB) + L1i L#122 (64KB) + Core L#122
PU L#244 (P#122)
PU L#245 (P#250)
L2 L#123 (2048KB) + L1d L#123 (48KB) + L1i L#123 (64KB) + Core L#123
PU L#246 (P#123)
PU L#247 (P#251)
L2 L#124 (2048KB) + L1d L#124 (48KB) + L1i L#124 (64KB) + Core L#124
PU L#248 (P#124)
PU L#249 (P#252)
L2 L#125 (2048KB) + L1d L#125 (48KB) + L1i L#125 (64KB) + Core L#125
PU L#250 (P#125)
PU L#251 (P#253)
L2 L#126 (2048KB) + L1d L#126 (48KB) + L1i L#126 (64KB) + Core L#126
PU L#252 (P#126)
PU L#253 (P#254)
L2 L#127 (2048KB) + L1d L#127 (48KB) + L1i L#127 (64KB) + Core L#127
PU L#254 (P#127)
PU L#255 (P#255)
HostBridge
PCI 81:00.0 (Co-Processor)
HostBridge
PCI 83:00.0 (Co-Processor)
HostBridge
PCI 8b:00.0 (Co-Processor)
HostBridge
PCI 8d:00.0 (Co-Processor)
HostBridge
PCIBridge
PCIBridge
PCIBridge
PCI 97:00.0 (Ethernet)
Net "eno7np0"
OpenFabrics "mlx5_14"
PCI 97:00.1 (Ethernet)
Net "enp151s0f1np1"
OpenFabrics "mlx5_15"
PCIBridge
PCIBridge
PCIBridge
PCI 9a:00.0 (3D)
CoProc(OpenCL) "opencl0d4"
HostBridge
PCIBridge
PCI a6:00.0 (NVMExp)
Block(Disk) "nvme0n1"
PCIBridge
PCI a7:00.0 (NVMExp)
Block(Disk) "nvme3n1"
HostBridge
PCIBridge
PCIBridge
PCIBridge
PCI b9:00.0 (Ethernet)
Net "eno9np0"
OpenFabrics "mlx5_16"
PCI b9:00.1 (Ethernet)
Net "enp185s0f1np1"
OpenFabrics "mlx5_17"
PCIBridge
PCIBridge
PCIBridge
PCI bc:00.0 (3D)
CoProc(OpenCL) "opencl0d5"
HostBridge
PCIBridge
PCIBridge
PCIBridge
PCIBridge
PCIBridge
PCI cc:00.0 (Ethernet)
Net "enp204s0f0np0"
OpenFabrics "mlx5_18"
PCI cc:00.1 (Ethernet)
Net "enp204s0f1np1"
OpenFabrics "mlx5_19"
PCIBridge
PCI cd:00.0 (NVMExp)
Block(Disk) "nvme4n1"
PCIBridge
PCI ce:00.0 (NVMExp)
Block(Disk) "nvme6n1"
PCIBridge
PCI cf:00.0 (NVMExp)
Block(Disk) "nvme8n1"
PCIBridge
PCI d0:00.0 (NVMExp)
Block(Disk) "nvme9n1"
HostBridge
PCIBridge
PCIBridge
PCIBridge
PCI dc:00.0 (Ethernet)
Net "eno8np0"
OpenFabrics "mlx5_20"
PCI dc:00.1 (Ethernet)
Net "enp220s0f1np1"
OpenFabrics "mlx5_21"
PCIBridge
PCIBridge
PCIBridge
PCI df:00.0 (3D)
CoProc(OpenCL) "opencl0d6"
HostBridge
PCIBridge
PCIBridge
PCIBridge
PCI ed:00.0 (Ethernet)
Net "eno6np0"
OpenFabrics "mlx5_22"
PCI ed:00.1 (Ethernet)
Net "enp237s0f1np1"
OpenFabrics "mlx5_23"
PCIBridge
PCIBridge
PCIBridge
PCI f0:00.0 (3D)
CoProc(OpenCL) "opencl0d7"

Example libvirt domain XML for B300 NVL8 8-GPU platform#

The following XML is the reference domain definition used on the tested B300 NVL8 system. The UUID, management MAC address, NVRAM path and disk paths have been replaced. CPU pinning, memory, controllers, host BDFs, hostdev grouping and QEMU MMIO settings are retained.

Use this XML as a reference. Validate it with the libvirt version on the target host, rediscover each source BDF, create site-specific storage and management settings, and review the inactive XML before the first boot.

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
<name>b300-ubuntu24-full-passthrough</name>
<uuid>REDACTED</uuid>

<memory unit='KiB'>1610612736</memory>
<currentMemory unit='KiB'>1610612736</currentMemory>
<memoryBacking>
<hugepages>
<page size='1048576' unit='KiB' nodeset='0'/>
<page size='1048576' unit='KiB' nodeset='1'/>
</hugepages>
<nosharepages/>
</memoryBacking>
<vcpu placement='static'>240</vcpu>
<iothreads>2</iothreads>
<cputune>
<vcpupin vcpu='0' cpuset='4'/>
<vcpupin vcpu='1' cpuset='132'/>
<vcpupin vcpu='2' cpuset='5'/>
<vcpupin vcpu='3' cpuset='133'/>
<vcpupin vcpu='4' cpuset='6'/>
<vcpupin vcpu='5' cpuset='134'/>
<vcpupin vcpu='6' cpuset='7'/>
<vcpupin vcpu='7' cpuset='135'/>
<vcpupin vcpu='8' cpuset='8'/>
<vcpupin vcpu='9' cpuset='136'/>
<vcpupin vcpu='10' cpuset='9'/>
<vcpupin vcpu='11' cpuset='137'/>
<vcpupin vcpu='12' cpuset='10'/>
<vcpupin vcpu='13' cpuset='138'/>
<vcpupin vcpu='14' cpuset='11'/>
<vcpupin vcpu='15' cpuset='139'/>
<vcpupin vcpu='16' cpuset='12'/>
<vcpupin vcpu='17' cpuset='140'/>
<vcpupin vcpu='18' cpuset='13'/>
<vcpupin vcpu='19' cpuset='141'/>
<vcpupin vcpu='20' cpuset='14'/>
<vcpupin vcpu='21' cpuset='142'/>
<vcpupin vcpu='22' cpuset='15'/>
<vcpupin vcpu='23' cpuset='143'/>
<vcpupin vcpu='24' cpuset='16'/>
<vcpupin vcpu='25' cpuset='144'/>
<vcpupin vcpu='26' cpuset='17'/>
<vcpupin vcpu='27' cpuset='145'/>
<vcpupin vcpu='28' cpuset='18'/>
<vcpupin vcpu='29' cpuset='146'/>
<vcpupin vcpu='30' cpuset='19'/>
<vcpupin vcpu='31' cpuset='147'/>
<vcpupin vcpu='32' cpuset='20'/>
<vcpupin vcpu='33' cpuset='148'/>
<vcpupin vcpu='34' cpuset='21'/>
<vcpupin vcpu='35' cpuset='149'/>
<vcpupin vcpu='36' cpuset='22'/>
<vcpupin vcpu='37' cpuset='150'/>
<vcpupin vcpu='38' cpuset='23'/>
<vcpupin vcpu='39' cpuset='151'/>
<vcpupin vcpu='40' cpuset='24'/>
<vcpupin vcpu='41' cpuset='152'/>
<vcpupin vcpu='42' cpuset='25'/>
<vcpupin vcpu='43' cpuset='153'/>
<vcpupin vcpu='44' cpuset='26'/>
<vcpupin vcpu='45' cpuset='154'/>
<vcpupin vcpu='46' cpuset='27'/>
<vcpupin vcpu='47' cpuset='155'/>
<vcpupin vcpu='48' cpuset='28'/>
<vcpupin vcpu='49' cpuset='156'/>
<vcpupin vcpu='50' cpuset='29'/>
<vcpupin vcpu='51' cpuset='157'/>
<vcpupin vcpu='52' cpuset='30'/>
<vcpupin vcpu='53' cpuset='158'/>
<vcpupin vcpu='54' cpuset='31'/>
<vcpupin vcpu='55' cpuset='159'/>
<vcpupin vcpu='56' cpuset='32'/>
<vcpupin vcpu='57' cpuset='160'/>
<vcpupin vcpu='58' cpuset='33'/>
<vcpupin vcpu='59' cpuset='161'/>
<vcpupin vcpu='60' cpuset='34'/>
<vcpupin vcpu='61' cpuset='162'/>
<vcpupin vcpu='62' cpuset='35'/>
<vcpupin vcpu='63' cpuset='163'/>
<vcpupin vcpu='64' cpuset='36'/>
<vcpupin vcpu='65' cpuset='164'/>
<vcpupin vcpu='66' cpuset='37'/>
<vcpupin vcpu='67' cpuset='165'/>
<vcpupin vcpu='68' cpuset='38'/>
<vcpupin vcpu='69' cpuset='166'/>
<vcpupin vcpu='70' cpuset='39'/>
<vcpupin vcpu='71' cpuset='167'/>
<vcpupin vcpu='72' cpuset='40'/>
<vcpupin vcpu='73' cpuset='168'/>
<vcpupin vcpu='74' cpuset='41'/>
<vcpupin vcpu='75' cpuset='169'/>
<vcpupin vcpu='76' cpuset='42'/>
<vcpupin vcpu='77' cpuset='170'/>
<vcpupin vcpu='78' cpuset='43'/>
<vcpupin vcpu='79' cpuset='171'/>
<vcpupin vcpu='80' cpuset='44'/>
<vcpupin vcpu='81' cpuset='172'/>
<vcpupin vcpu='82' cpuset='45'/>
<vcpupin vcpu='83' cpuset='173'/>
<vcpupin vcpu='84' cpuset='46'/>
<vcpupin vcpu='85' cpuset='174'/>
<vcpupin vcpu='86' cpuset='47'/>
<vcpupin vcpu='87' cpuset='175'/>
<vcpupin vcpu='88' cpuset='48'/>
<vcpupin vcpu='89' cpuset='176'/>
<vcpupin vcpu='90' cpuset='49'/>
<vcpupin vcpu='91' cpuset='177'/>
<vcpupin vcpu='92' cpuset='50'/>
<vcpupin vcpu='93' cpuset='178'/>
<vcpupin vcpu='94' cpuset='51'/>
<vcpupin vcpu='95' cpuset='179'/>
<vcpupin vcpu='96' cpuset='52'/>
<vcpupin vcpu='97' cpuset='180'/>
<vcpupin vcpu='98' cpuset='53'/>
<vcpupin vcpu='99' cpuset='181'/>
<vcpupin vcpu='100' cpuset='54'/>
<vcpupin vcpu='101' cpuset='182'/>
<vcpupin vcpu='102' cpuset='55'/>
<vcpupin vcpu='103' cpuset='183'/>
<vcpupin vcpu='104' cpuset='56'/>
<vcpupin vcpu='105' cpuset='184'/>
<vcpupin vcpu='106' cpuset='57'/>
<vcpupin vcpu='107' cpuset='185'/>
<vcpupin vcpu='108' cpuset='58'/>
<vcpupin vcpu='109' cpuset='186'/>
<vcpupin vcpu='110' cpuset='59'/>
<vcpupin vcpu='111' cpuset='187'/>
<vcpupin vcpu='112' cpuset='60'/>
<vcpupin vcpu='113' cpuset='188'/>
<vcpupin vcpu='114' cpuset='61'/>
<vcpupin vcpu='115' cpuset='189'/>
<vcpupin vcpu='116' cpuset='62'/>
<vcpupin vcpu='117' cpuset='190'/>
<vcpupin vcpu='118' cpuset='63'/>
<vcpupin vcpu='119' cpuset='191'/>
<vcpupin vcpu='120' cpuset='68'/>
<vcpupin vcpu='121' cpuset='196'/>
<vcpupin vcpu='122' cpuset='69'/>
<vcpupin vcpu='123' cpuset='197'/>
<vcpupin vcpu='124' cpuset='70'/>
<vcpupin vcpu='125' cpuset='198'/>
<vcpupin vcpu='126' cpuset='71'/>
<vcpupin vcpu='127' cpuset='199'/>
<vcpupin vcpu='128' cpuset='72'/>
<vcpupin vcpu='129' cpuset='200'/>
<vcpupin vcpu='130' cpuset='73'/>
<vcpupin vcpu='131' cpuset='201'/>
<vcpupin vcpu='132' cpuset='74'/>
<vcpupin vcpu='133' cpuset='202'/>
<vcpupin vcpu='134' cpuset='75'/>
<vcpupin vcpu='135' cpuset='203'/>
<vcpupin vcpu='136' cpuset='76'/>
<vcpupin vcpu='137' cpuset='204'/>
<vcpupin vcpu='138' cpuset='77'/>
<vcpupin vcpu='139' cpuset='205'/>
<vcpupin vcpu='140' cpuset='78'/>
<vcpupin vcpu='141' cpuset='206'/>
<vcpupin vcpu='142' cpuset='79'/>
<vcpupin vcpu='143' cpuset='207'/>
<vcpupin vcpu='144' cpuset='80'/>
<vcpupin vcpu='145' cpuset='208'/>
<vcpupin vcpu='146' cpuset='81'/>
<vcpupin vcpu='147' cpuset='209'/>
<vcpupin vcpu='148' cpuset='82'/>
<vcpupin vcpu='149' cpuset='210'/>
<vcpupin vcpu='150' cpuset='83'/>
<vcpupin vcpu='151' cpuset='211'/>
<vcpupin vcpu='152' cpuset='84'/>
<vcpupin vcpu='153' cpuset='212'/>
<vcpupin vcpu='154' cpuset='85'/>
<vcpupin vcpu='155' cpuset='213'/>
<vcpupin vcpu='156' cpuset='86'/>
<vcpupin vcpu='157' cpuset='214'/>
<vcpupin vcpu='158' cpuset='87'/>
<vcpupin vcpu='159' cpuset='215'/>
<vcpupin vcpu='160' cpuset='88'/>
<vcpupin vcpu='161' cpuset='216'/>
<vcpupin vcpu='162' cpuset='89'/>
<vcpupin vcpu='163' cpuset='217'/>
<vcpupin vcpu='164' cpuset='90'/>
<vcpupin vcpu='165' cpuset='218'/>
<vcpupin vcpu='166' cpuset='91'/>
<vcpupin vcpu='167' cpuset='219'/>
<vcpupin vcpu='168' cpuset='92'/>
<vcpupin vcpu='169' cpuset='220'/>
<vcpupin vcpu='170' cpuset='93'/>
<vcpupin vcpu='171' cpuset='221'/>
<vcpupin vcpu='172' cpuset='94'/>
<vcpupin vcpu='173' cpuset='222'/>
<vcpupin vcpu='174' cpuset='95'/>
<vcpupin vcpu='175' cpuset='223'/>
<vcpupin vcpu='176' cpuset='96'/>
<vcpupin vcpu='177' cpuset='224'/>
<vcpupin vcpu='178' cpuset='97'/>
<vcpupin vcpu='179' cpuset='225'/>
<vcpupin vcpu='180' cpuset='98'/>
<vcpupin vcpu='181' cpuset='226'/>
<vcpupin vcpu='182' cpuset='99'/>
<vcpupin vcpu='183' cpuset='227'/>
<vcpupin vcpu='184' cpuset='100'/>
<vcpupin vcpu='185' cpuset='228'/>
<vcpupin vcpu='186' cpuset='101'/>
<vcpupin vcpu='187' cpuset='229'/>
<vcpupin vcpu='188' cpuset='102'/>
<vcpupin vcpu='189' cpuset='230'/>
<vcpupin vcpu='190' cpuset='103'/>
<vcpupin vcpu='191' cpuset='231'/>
<vcpupin vcpu='192' cpuset='104'/>
<vcpupin vcpu='193' cpuset='232'/>
<vcpupin vcpu='194' cpuset='105'/>
<vcpupin vcpu='195' cpuset='233'/>
<vcpupin vcpu='196' cpuset='106'/>
<vcpupin vcpu='197' cpuset='234'/>
<vcpupin vcpu='198' cpuset='107'/>
<vcpupin vcpu='199' cpuset='235'/>
<vcpupin vcpu='200' cpuset='108'/>
<vcpupin vcpu='201' cpuset='236'/>
<vcpupin vcpu='202' cpuset='109'/>
<vcpupin vcpu='203' cpuset='237'/>
<vcpupin vcpu='204' cpuset='110'/>
<vcpupin vcpu='205' cpuset='238'/>
<vcpupin vcpu='206' cpuset='111'/>
<vcpupin vcpu='207' cpuset='239'/>
<vcpupin vcpu='208' cpuset='112'/>
<vcpupin vcpu='209' cpuset='240'/>
<vcpupin vcpu='210' cpuset='113'/>
<vcpupin vcpu='211' cpuset='241'/>
<vcpupin vcpu='212' cpuset='114'/>
<vcpupin vcpu='213' cpuset='242'/>
<vcpupin vcpu='214' cpuset='115'/>
<vcpupin vcpu='215' cpuset='243'/>
<vcpupin vcpu='216' cpuset='116'/>
<vcpupin vcpu='217' cpuset='244'/>
<vcpupin vcpu='218' cpuset='117'/>
<vcpupin vcpu='219' cpuset='245'/>
<vcpupin vcpu='220' cpuset='118'/>
<vcpupin vcpu='221' cpuset='246'/>
<vcpupin vcpu='222' cpuset='119'/>
<vcpupin vcpu='223' cpuset='247'/>
<vcpupin vcpu='224' cpuset='120'/>
<vcpupin vcpu='225' cpuset='248'/>
<vcpupin vcpu='226' cpuset='121'/>
<vcpupin vcpu='227' cpuset='249'/>
<vcpupin vcpu='228' cpuset='122'/>
<vcpupin vcpu='229' cpuset='250'/>
<vcpupin vcpu='230' cpuset='123'/>
<vcpupin vcpu='231' cpuset='251'/>
<vcpupin vcpu='232' cpuset='124'/>
<vcpupin vcpu='233' cpuset='252'/>
<vcpupin vcpu='234' cpuset='125'/>
<vcpupin vcpu='235' cpuset='253'/>
<vcpupin vcpu='236' cpuset='126'/>
<vcpupin vcpu='237' cpuset='254'/>
<vcpupin vcpu='238' cpuset='127'/>
<vcpupin vcpu='239' cpuset='255'/>
<emulatorpin cpuset='0-3,64-67,128-131,192-195'/>
<iothreadpin iothread='1' cpuset='0-3,128-131'/>
<iothreadpin iothread='2' cpuset='64-67,192-195'/>
</cputune>
<numatune>
<memory mode='strict' nodeset='0-1'/>
<memnode cellid='0' mode='strict' nodeset='0'/>
<memnode cellid='1' mode='strict' nodeset='1'/>
</numatune>
<resource>
<partition>/machine</partition>
</resource>
<os firmware='efi'>
<type arch='x86_64' machine='pc-q35-9.2'>hvm</type>
<firmware>
<feature enabled='no' name='enrolled-keys'/>
<feature enabled='no' name='secure-boot'/>
</firmware>
<loader readonly='yes' type='pflash' format='raw'>/usr/share/OVMF/OVMF_CODE_4M.fd</loader>
<nvram template='/usr/share/OVMF/OVMF_VARS_4M.fd' templateFormat='raw' format='raw'>/var/lib/libvirt/qemu/nvram/b300-guest_VARS.fd</nvram>
<boot dev='hd'/>
</os>
<features>
<acpi/>
<apic/>
<vmport state='off'/>
</features>
<cpu mode='host-passthrough' check='none' migratable='off'>
<topology sockets='2' dies='1' clusters='1' cores='60' threads='2'/>
<maxphysaddr mode='passthrough'/>
<numa>
<cell id='0' cpus='0-119' memory='805306368' unit='KiB'/>
<cell id='1' cpus='120-239' memory='805306368' unit='KiB'/>
</numa>
</cpu>
<clock offset='utc'/>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>preserve</on_crash>
<pm>
<suspend-to-mem enabled='no'/>
<suspend-to-disk enabled='no'/>
</pm>
<devices>
<emulator>/usr/bin/qemu-system-x86_64</emulator>
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2' cache='none' io='native' discard='unmap'/>
<source file='/var/lib/libvirt/images/b300-guest.qcow2'/>
<target dev='vda' bus='virtio'/>
<address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
</disk>
<disk type='file' device='cdrom'>
<driver name='qemu' type='raw'/>
<target dev='sda' bus='sata'/>
<readonly/>
<address type='drive' controller='0' bus='0' target='0' unit='0'/>
</disk>
<controller type='sata' index='0'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
</controller>
<controller type='pci' index='0' model='pcie-root'/>
<controller type='pci' index='1' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='1' port='0x11'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0' multifunction='on'/>
</controller>
<controller type='pci' index='2' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='2' port='0x12'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
</controller>
<controller type='pci' index='3' model='pcie-expander-bus'>
<model name='pxb-pcie'/>
<target busNr='32'>
<node>0</node>
</target>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0' multifunction='on'/>
</controller>
<controller type='pci' index='4' model='pcie-expander-bus'>
<model name='pxb-pcie'/>
<target busNr='128'>
<node>1</node>
</target>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x1'/>
</controller>
<controller type='pci' index='5' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='5' port='0x15'/>
<address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0' multifunction='on'/>
</controller>
<controller type='pci' index='6' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='6' port='0x16'/>
<address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x1'/>
</controller>
<controller type='pci' index='7' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='7' port='0x17'/>
<address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x2'/>
</controller>
<controller type='pci' index='8' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='8' port='0x18'/>
<address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x3'/>
</controller>
<controller type='pci' index='9' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='9' port='0x19'/>
<address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x4'/>
</controller>
<controller type='pci' index='10' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='10' port='0x1a'/>
<address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0' multifunction='on'/>
</controller>
<controller type='pci' index='11' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='11' port='0x1b'/>
<address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x1'/>
</controller>
<controller type='pci' index='12' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='12' port='0x1c'/>
<address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x2'/>
</controller>
<controller type='pci' index='13' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='13' port='0x1d'/>
<address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x3'/>
</controller>
<controller type='pci' index='14' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='40' port='0xa'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
</controller>
<controller type='pci' index='15' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='41' port='0xb'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x3'/>
</controller>
<controller type='pci' index='16' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='42' port='0xc'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x4'/>
</controller>
<controller type='usb' index='0' model='qemu-xhci'>
<address type='pci' domain='0x0000' bus='0x0e' slot='0x00' function='0x0'/>
</controller>
<controller type='virtio-serial' index='0'>
<address type='pci' domain='0x0000' bus='0x0f' slot='0x00' function='0x0'/>
</controller>
<interface type='network'>
<mac address='52:54:00:XX:XX:XX'/>
<source network='default'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
</interface>
<serial type='pty'>
<target type='isa-serial' port='0'>
<model name='isa-serial'/>
</target>
</serial>
<console type='pty'>
<target type='serial' port='0'/>
</console>
<channel type='unix'>
<target type='virtio' name='org.qemu.guest_agent.0'/>
<address type='virtio-serial' controller='0' bus='0' port='1'/>
</channel>
<input type='mouse' bus='ps2'/>
<input type='keyboard' bus='ps2'/>
<audio id='1' type='none'/>
<video>
<model type='none'/>
</video>
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='vfio'/>
<source>
<address domain='0x0000' bus='0x1a' slot='0x00' function='0x0'/>
</source>
<rom bar='off'/>
<address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0' multifunction='on'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='vfio'/>
<source>
<address domain='0x0000' bus='0x17' slot='0x00' function='0x0'/>
</source>
<rom bar='off'/>
<address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x1'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='vfio'/>
<source>
<address domain='0x0000' bus='0x17' slot='0x00' function='0x1'/>
</source>
<rom bar='off'/>
<address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x2'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='vfio'/>
<source>
<address domain='0x0000' bus='0x3c' slot='0x00' function='0x0'/>
</source>
<rom bar='off'/>
<address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0' multifunction='on'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='vfio'/>
<source>
<address domain='0x0000' bus='0x39' slot='0x00' function='0x0'/>
</source>
<rom bar='off'/>
<address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x1'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='vfio'/>
<source>
<address domain='0x0000' bus='0x39' slot='0x00' function='0x1'/>
</source>
<rom bar='off'/>
<address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x2'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='vfio'/>
<source>
<address domain='0x0000' bus='0x62' slot='0x00' function='0x0'/>
</source>
<rom bar='off'/>
<address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0' multifunction='on'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='vfio'/>
<source>
<address domain='0x0000' bus='0x5f' slot='0x00' function='0x0'/>
</source>
<rom bar='off'/>
<address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x1'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='vfio'/>
<source>
<address domain='0x0000' bus='0x5f' slot='0x00' function='0x1'/>
</source>
<rom bar='off'/>
<address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x2'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='vfio'/>
<source>
<address domain='0x0000' bus='0x73' slot='0x00' function='0x0'/>
</source>
<rom bar='off'/>
<address type='pci' domain='0x0000' bus='0x08' slot='0x00' function='0x0' multifunction='on'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='vfio'/>
<source>
<address domain='0x0000' bus='0x70' slot='0x00' function='0x0'/>
</source>
<rom bar='off'/>
<address type='pci' domain='0x0000' bus='0x08' slot='0x00' function='0x1'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='vfio'/>
<source>
<address domain='0x0000' bus='0x70' slot='0x00' function='0x1'/>
</source>
<rom bar='off'/>
<address type='pci' domain='0x0000' bus='0x08' slot='0x00' function='0x2'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='vfio'/>
<source>
<address domain='0x0000' bus='0x9a' slot='0x00' function='0x0'/>
</source>
<rom bar='off'/>
<address type='pci' domain='0x0000' bus='0x0a' slot='0x00' function='0x0' multifunction='on'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='vfio'/>
<source>
<address domain='0x0000' bus='0x97' slot='0x00' function='0x0'/>
</source>
<rom bar='off'/>
<address type='pci' domain='0x0000' bus='0x0a' slot='0x00' function='0x1'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='vfio'/>
<source>
<address domain='0x0000' bus='0x97' slot='0x00' function='0x1'/>
</source>
<rom bar='off'/>
<address type='pci' domain='0x0000' bus='0x0a' slot='0x00' function='0x2'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='vfio'/>
<source>
<address domain='0x0000' bus='0xbc' slot='0x00' function='0x0'/>
</source>
<rom bar='off'/>
<address type='pci' domain='0x0000' bus='0x0b' slot='0x00' function='0x0' multifunction='on'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='vfio'/>
<source>
<address domain='0x0000' bus='0xb9' slot='0x00' function='0x0'/>
</source>
<rom bar='off'/>
<address type='pci' domain='0x0000' bus='0x0b' slot='0x00' function='0x1'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='vfio'/>
<source>
<address domain='0x0000' bus='0xb9' slot='0x00' function='0x1'/>
</source>
<rom bar='off'/>
<address type='pci' domain='0x0000' bus='0x0b' slot='0x00' function='0x2'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='vfio'/>
<source>
<address domain='0x0000' bus='0xdf' slot='0x00' function='0x0'/>
</source>
<rom bar='off'/>
<address type='pci' domain='0x0000' bus='0x0c' slot='0x00' function='0x0' multifunction='on'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='vfio'/>
<source>
<address domain='0x0000' bus='0xdc' slot='0x00' function='0x0'/>
</source>
<rom bar='off'/>
<address type='pci' domain='0x0000' bus='0x0c' slot='0x00' function='0x1'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='vfio'/>
<source>
<address domain='0x0000' bus='0xdc' slot='0x00' function='0x1'/>
</source>
<rom bar='off'/>
<address type='pci' domain='0x0000' bus='0x0c' slot='0x00' function='0x2'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='vfio'/>
<source>
<address domain='0x0000' bus='0xf0' slot='0x00' function='0x0'/>
</source>
<rom bar='off'/>
<address type='pci' domain='0x0000' bus='0x0d' slot='0x00' function='0x0' multifunction='on'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='vfio'/>
<source>
<address domain='0x0000' bus='0xed' slot='0x00' function='0x0'/>
</source>
<rom bar='off'/>
<address type='pci' domain='0x0000' bus='0x0d' slot='0x00' function='0x1'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='vfio'/>
<source>
<address domain='0x0000' bus='0xed' slot='0x00' function='0x1'/>
</source>
<rom bar='off'/>
<address type='pci' domain='0x0000' bus='0x0d' slot='0x00' function='0x2'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='vfio'/>
<source>
<address domain='0x0000' bus='0x27' slot='0x00' function='0x0'/>
</source>
<rom bar='off'/>
<address type='pci' domain='0x0000' bus='0x09' slot='0x00' function='0x0' multifunction='on'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='vfio'/>
<source>
<address domain='0x0000' bus='0x27' slot='0x00' function='0x1'/>
</source>
<rom bar='off'/>
<address type='pci' domain='0x0000' bus='0x09' slot='0x00' function='0x1'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='vfio'/>
<source>
<address domain='0x0000' bus='0x27' slot='0x00' function='0x2'/>
</source>
<rom bar='off'/>
<address type='pci' domain='0x0000' bus='0x09' slot='0x00' function='0x2'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='vfio'/>
<source>
<address domain='0x0000' bus='0x27' slot='0x00' function='0x3'/>
</source>
<rom bar='off'/>
<address type='pci' domain='0x0000' bus='0x09' slot='0x00' function='0x3'/>
</hostdev>
<watchdog model='itco' action='reset'/>
<memballoon model='none'/>
<rng model='virtio'>
<backend model='random'>/dev/urandom</backend>
<address type='pci' domain='0x0000' bus='0x10' slot='0x00' function='0x0'/>
</rng>
</devices>
<qemu:commandline>
<qemu:arg value='-fw_cfg'/>
<qemu:arg value='name=opt/ovmf/X-PciMmio64Mb,string=16777216'/>
<qemu:arg value='-global'/>
<qemu:arg value='q35-pcihost.pci-hole64-size=16384G'/>
</qemu:commandline>
</domain>

Domain XML syntax: libvirt Domain XML format.

Platform hardware: NVIDIA B300 NVL8 System User Guide.