12. NetApp E-series BeeGFS with GDS Solution Deployment#
NetApp supports BeeGFS High Availability.
Refer to the BeeGFS with Netapp E-Series Technical Report on how to deploy the BeeGFS parallel file system: Netapp BeeGFS Deployment. For deployments requiring high availability, refer to BeeGFS High Availability with NetApp E-Series.
12.1. Netapp BeeGFS/GPUDirect Storage and Package Requirements#
BeeGFS client and storage with GDS:
CUDA and GDS are only required on the beegfs-client hosts. There are no CUDA or GPUDirect Storage requirements for the BeeGFS server hosts.
12.2. BeeGFS Client Configuration for GDS#
After installing beegfs-client, the client build needs to be configured for RDMA and GDS.
Edit
/etc/beegfs/beegfs-client-autobuild.conf. Change line 57 of the file to:buildArgs=-j8 NVFS_H_PATH=/usr/src/mlnx-ofed-kernel-5.4/drivers/nvme/host OFED_INCLUDE_PATH=/usr/src/ofa_kernel/default/include
This should all be on the same line.
Rebuild beegfs-client:
sudo /etc/init.d/beegfs-client rebuild
12.3. GPU/HCA Topology on the Client - DGX-A100 and OSS servers Client Server#
Client Server:
ibdev |
netdev |
IP |
GPU |
Numa |
OSS |
Target |
Mount Point |
|---|---|---|---|---|---|---|---|
mlx5_4 |
ibp97s0f0 |
10.10.0.177/24 |
0,1,2,3 |
0 |
meta 1 |
5,6,7,8 |
|
mlx5_5 |
ibp97s0f1 |
10.10.1.177/24 |
0,1,2,3 |
0 |
meta 1 |
5,6,7,8 |
|
mlx5_10 |
ibp225s0f0 |
10.10.2.157/24 |
4,5,6,7 |
4 |
meta 2 |
1,2,3,4 |
|
mlx5_11 |
ibp225s0f1 |
10.10.3.157/24 |
4,5,6,7 |
4 |
meta 2 |
1,2,3,4 |
|
OSS Servers:
OSS |
ID |
IP |
Numa |
|---|---|---|---|
meta01-numa0-1 |
1001 |
10.10.0.131:8003 |
0 |
meta01-numa1-2 |
1002 |
10.10.1.131:8004 |
1 |
meta02-numa0-1 |
2001 |
10.10.2.132:8003 |
0 |
meta02-numa1-2 |
2002 |
10.10.3.132:8004 |
1 |
12.4. Verify the Setup#
To verify the setup, run the following commands on any client:
12.4.1. List the Management Node#
root@dgxa100-b:/sys/class# beegfs-ctl --listnodes --nodetype=management --details
meta-02.cpoc.local [ID: 1]
Ports: UDP: 8008; TCP: 8008
Interfaces: em3(TCP)
12.4.2. List the Metadata Nodes#
root@dgxa100-b:/sys/class# beegfs-ctl --listnodes --nodetype=meta -details
meta01-numa0-1-meta [ID: 1101]
Ports: UDP: 8005; TCP: 8005
Interfaces: ib0:net1(RDMA) ib0:net1(TCP)
meta01-numa1-2-meta [ID: 1102]
Ports: UDP: 8006; TCP: 8006
Interfaces: ib2:net3(RDMA) ib2:net3(TCP)
meta02-numa0-1-meta [ID: 2101]
Ports: UDP: 8005; TCP: 8005
Interfaces: ib0:net0(RDMA) ib0:net0(TCP)
meta02-numa1-2-meta [ID: 2102]
Ports: UDP: 8006; TCP: 8006
Interfaces: ib2:net2(RDMA) ib2:net2(TCP)
Number of nodes: 4
Root: 2101
12.4.3. List the Storage Nodes#
root@dgxa100-b:/sys/class# beegfs-ctl --listnodes --nodetype=storage -details
meta01-numa0-1 [ID: 1001]
Ports: UDP: 8003; TCP: 8003
Interfaces: ib0:net1(RDMA) ib0:net1(TCP)
meta01-numa1-2 [ID: 1002]
Ports: UDP: 8004; TCP: 8004
Interfaces: ib2:net3(RDMA) ib2:net3(TCP)
meta02-numa0-1 [ID: 2001]
Ports: UDP: 8003; TCP: 8003
Interfaces: ib0:net0(RDMA) ib0:net0(TCP)
meta02-numa1-2 [ID: 2002]
Ports: UDP: 8004; TCP: 8004
Interfaces: ib2:net2(RDMA) ib2:net2(TCP)
Number of nodes: 4
12.4.4. List the Client Nodes#
root@dgxa100-b:/sys/class# beegfs-ctl --listnodes --nodetype=client --details
B4330-6161F689-dgxa100-b [ID: 11]
Ports: UDP: 8004; TCP: 0
Interfaces: ibp97s0f0(RDMA) ibp97s0f0(TCP) ibp97s0f1(TCP) ibp97s0f1(RDMA) ibp225s0f0(TCP)
ibp225s0f0(RDMA) ibp225s0f1(TCP) ibp225s0f1(RDMA)
12.4.5. Display Client Connections#
root@dgxa100-b:/sys/class# beegfs-net
mgmt_nodes
=============
meta-02.cpoc.local [ID: 1]
Connections: TCP: 1 (192.168.0.132:8008);
meta_nodes
=============
meta01-numa0-1-meta [ID: 1101]
Connections: RDMA: 1 (10.10.1.131:8005);
meta01-numa1-2-meta [ID: 1102]
Connections: RDMA: 1 (10.10.3.131:8006);
meta02-numa0-1-meta [ID: 2101]
Connections: RDMA: 1 (10.10.0.132:8005);
meta02-numa1-2-meta [ID: 2102]
Connections: RDMA: 1 (10.10.2.132:8006);
storage_nodes
=============
meta01-numa0-1 [ID: 1001]
Connections: RDMA: 8 (10.10.1.131:8003);
meta01-numa1-2 [ID: 1002]
Connections: RDMA: 8 (10.10.3.131:8004);
meta02-numa0-1 [ID: 2001]
Connections: RDMA: 16 (10.10.0.132:8003);
meta02-numa1-2 [ID: 2002]
Connections: RDMA: 8 (10.10.2.132:8004);
12.4.6. Verify Connectivity to the Different Services#
root@dgxa100-b:/sys/class# beegfs-check-servers
Management
==========
meta-02.cpoc.local [ID: 1]: reachable at 192.168.0.132:8008 (protocol: TCP)
Metadata
==========
meta01-numa0-1-meta [ID: 1101]: reachable at 10.10.1.131:8005 (protocol: TCP)
meta01-numa1-2-meta [ID: 1102]: reachable at 10.10.3.131:8006 (protocol: TCP)
meta02-numa0-1-meta [ID: 2101]: reachable at 10.10.0.132:8005 (protocol: TCP)
meta02-numa1-2-meta [ID: 2102]: reachable at 10.10.2.132:8006 (protocol: TCP)
Storage
==========
meta01-numa0-1 [ID: 1001]: reachable at 10.10.1.131:8003 (protocol: TCP)
meta01-numa1-2 [ID: 1002]: reachable at 10.10.3.131:8004 (protocol: TCP)
meta02-numa0-1 [ID: 2001]: reachable at 10.10.0.132:8003 (protocol: TCP)
meta02-numa1-2 [ID: 2002]: reachable at 10.10.2.132:8004 (protocol: TCP)
12.4.7. List Storage Pools#
In this example we used the default mounting point:
root@dgxa100-b:/sys/class# sudo beegfs-ctl -liststoragepools
Pool ID Pool Description Targets Buddy Groups
======= ================== ============================ ============================
1 Default 1,2,3,4,5,6,7,8
12.4.8. Display the Free Space and inodes on the Storage and Metadata Targets#
root@dgxa100-b:/sys/class# beegfs-df
METADATA SERVERS:
TargetID Cap. Pool Total Free % ITotal IFree %
======== ========= ===== ==== = ====== ===== =
1101 normal 573.3GiB 572.9GiB 100% 401.1M 401.0M 100%
1102 normal 573.3GiB 572.9GiB 100% 401.1M 401.0M 100%
2101 normal 573.3GiB 572.9GiB 100% 401.1M 401.0M 100%
2102 normal 573.3GiB 572.9GiB 100% 401.1M 401.0M 100%
STORAGE TARGETS:
TargetID Cap. Pool Total Free % ITotal IFree %
======== ========= ===== ==== = ====== ===== =
1 normal 2574.7GiB 1470.8GiB 57% 270.1M 270.1M 100%
2 normal 2574.7GiB 1404.0GiB 55% 270.1M 270.1M 100%
3 normal 2574.7GiB 1265.5GiB 49% 270.1M 270.1M 100%
4 normal 2574.7GiB 1278.5GiB 50% 270.1M 270.1M 100%
5 normal 2574.7GiB 1274.0GiB 49% 270.1M 270.1M 100%
6 normal 2574.7GiB 1342.6GiB 52% 270.1M 270.1M 100%
7 normal 2574.7GiB 1485.3GiB 58% 270.1M 270.1M 100%
8 normal 2574.7GiB 1481.7GiB 58% 270.1M 270.1M 100%
12.5. Testing#
12.5.1. Verifying Integration is Working#
Once beegfs-client has been started with GDS support, a basic test can be performed to verify that the integration is working:
root@dgxa100-b:/usr/local/cuda-11.4/gds/tools# ./gdscheck.py -p
GDS release version: 1.16.0.48
nvidia_fs version: 2.27 nvidia_fs minimum version: 2.12
Platform: x86_64
============
ENVIRONMENT:
============
=====================
DRIVER CONFIGURATION:
=====================
…
BeeGFS : nvfs, compat
…
Cuda Driver Version Installed: 13010
Platform: ProLiant DL360 Gen10, Arch: x86_64(Linux 5.14.0-570.25.1.el9_6.x86_64)
Platform verification succeeded
12.5.2. Conducting a Basic NetApp BeeGFS File System Test#
/usr/local/cuda/gds/tools/gdsio_verify -f /mnt/beegfs/file 1g -d 0 -o 0 -s 1G -n 1 -m 1
gpu index :0, file :/mnt/beegfs/file 1g, gpu buffer alignment :0, gpu buffer offset :0, gpu devptr offset :0, file offset :0, io_requested :1073741824, io_chunk_size :1073741824, bufregister :true, sync :1, nr ios :1, fsync :0,
Data Verification Success