8. Understanding EXAScaler File System Performance#
Depending on the type of host channel adapter (HCA), commonly known as a NIC, there are mod parameters that can be tuned for LNet. The NICs that you select should be up and healthy.
To verify the health by mounting and running some basic tests, use lnetctl health statistics, and
run the following command:
$ cat /etc/modprobe.d/lustre.conf
Example output:
options libcfs cpu_npartitions=24 cpu_pattern=""
options lnet networks="o2ib0(ib1,ib2,ib3,ib4,ib6,ib7,ib8,ib9)"
options ko2iblnd peer_credits=32 concurrent_sends=64 peer_credits_hiw=16 map_on_demand=0
8.1. osc Tuning Performance Parameters#
The following is information about tuning file system parameters.
Note
To maximize the throughput, you can tune the following EXAScaler file system client parameters, based on the network.
Run the following command:
$ lctl get_param osc.*.max* osc.*.checksums
Review the output, for example:
$ lctl get_param osc.*.max* osc.*.checksums
osc.ai400-OST0024-osc-ffff916f6533a000.max_pages_per_rpc=4096
osc.ai400-OST0024-osc-ffff916f6533a000.max_dirty_mb=512
osc.ai400-OST0024-osc-ffff916f6533a000.max_rpcs_in_flight=32
osc.ai400-OST0024-osc-ffff916f6533a000.checksums=0
To check llite parameters, run $ lctl get_param llite.*.*.
8.2. Miscellaneous Commands for osc, mdc, and stripesize#
If the tuning parameters are set correctly, you can use these parameters to observe.
To get an overall EXAScaler file system client side statistics, run the following command:
$ lctl get_param osc.*.import
Note
The command includes
rpcinformation.Review the output, for example:
$ watch -d 'lctl get_param osc.*.import | grep -B 1 inflight' rpcs: inflight: 5 rpcs: inflight: 33To get the maximum number of pages that can be transferred per rpc in a EXAScaler file system client, run the following command:
$ lctl get_param osc.*.max_pages_per_rpc
To get the overall rpc statistics from a EXAScaler file system client, run the following command:
$ lctl set_param osc.*.rpc_stats=clear (to reset osc stats) $ lctl get_param osc.*.rpc_stats
Review the output, for example:
osc.ai200-OST0000-osc-ffff8e0b47c73800.rpc_stats= snapshot_time: 1589919461.185215594 (secs.nsecs) read RPCs in flight: 0 write RPCs in flight: 0 pending write pages: 0 pending read pages: 0 read write pages per rpc rpcs % cum % | rpcs % cum % 1: 14222350 77 77 | 0 0 0 2: 0 0 77 | 0 0 0 4: 0 0 77 | 0 0 0 8: 0 0 77 | 0 0 0 16: 0 0 77 | 0 0 0 32: 0 0 77 | 0 0 0 64: 0 0 77 | 0 0 0 128: 0 0 77 | 0 0 0 256: 4130365 22 100 | 0 0 0 read write rpcs in flight rpcs % cum % | rpcs % cum % 0: 0 0 0 | 0 0 0 1: 3236263 17 17 | 0 0 0 2: 117001 0 18 | 0 0 0 3: 168119 0 19 | 0 0 0 4: 153295 0 20 | 0 0 0 5: 91598 0 20 | 0 0 0 6: 42476 0 20 | 0 0 0 7: 17578 0 20 | 0 0 0 8: 9454 0 20 | 0 0 0 9: 7611 0 20 | 0 0 0 10: 7772 0 20 | 0 0 0 11: 8914 0 21 | 0 0 0 12: 9350 0 21 | 0 0 0 13: 8559 0 21 | 0 0 0 14: 8734 0 21 | 0 0 0 15: 10784 0 21 | 0 0 0 16: 11386 0 21 | 0 0 0 17: 13148 0 21 | 0 0 0 18: 15473 0 21 | 0 0 0 19: 17619 0 21 | 0 0 0 20: 18851 0 21 | 0 0 0 21: 21853 0 21 | 0 0 0 22: 21236 0 21 | 0 0 0 23: 21588 0 22 | 0 0 0 24: 23859 0 22 | 0 0 0 25: 24049 0 22 | 0 0 0 26: 26232 0 22 | 0 0 0 27: 29853 0 22 | 0 0 0 28: 31992 0 22 | 0 0 0 29: 43626 0 22 | 0 0 0 30: 116116 0 23 | 0 0 0 31: 14018326 76 100 | 0 0 0
To get statistics that are related to client metadata operations, run the following command:
Note
MetaDataClient (MDC) is the client side counterpart of MetaData Server (MDS).
$ lctl get_param mdc.*.md_stats
To get the stripe layout of the file on the EXAScaler file system, run the following command:
$ lfs getstripe /mnt/ai200
8.3. Getting the Number of Configured Object-Based Disks#
To get the number of configured object-bsaed disks:
Run the following command:
$ lctl get_param lov.*.target_obd
Review the output, for example:
0: ai200-OST0000_UUID ACTIVE 1: ai200-OST0001_UUID ACTIVE
8.5. Getting Metadata Statistics#
To get metadata statistics:
Run the following command:
$ lctl get_param lmv.*.md_stats
Review the output, for example:
snapshot_time 1571271931.653827773 secs.nsecs close 8 samples [reqs] create 1 samples [reqs] getattr 1 samples [reqs] intent_lock 81 samples[reqs] read_page 3 samples [reqs] revalidate_lock 1 samples [reqs]
8.6. Checking for an Existing Mount#
To check for an existing mount in the EXAScaler file system:
Run the following command:
$ mount | grep lustre
Review the output, for example:
192.168.1.61@o2ib,192.168.1.62@o2ib1:/ai200 on /mnt/ai200 type lustre (rw,flock,lazystatfs)
8.7. Unmounting an EXAScaler File System Cluster#
To unmount an EXAScaler file system cluster, run the following command:
$ sudo umount /mnt/ai200
8.8. Getting a Summary of EXAScaler File System Statistics#
You can get a summary of statistics for the EXAScaler file system.
Refer to the Lustre Monitoring and Statistics Guide for more information.
8.9. Using GPUDirect Storage in Poll Mode#
This section describes how to use GDS in Poll Mode with EXAScaler file system files that have a Stripe Count greater than 1.
Currently, if poll mode is enabled, cuFileReads or cuFileWrites might return bytes that are less than the
bytes that were requested. This behavior is POSIX compliant and is observed with files that have a stripe count
that is greater than the count in their layout. If behavior occurs, we recommend that the application checks for
returned bytes and continues until all of the data is consumed. You can also set the corresponding
properties.poll_mode_max_size_kb,(say 1024(KB)) value to the lowest possible stripe size in the directory.
This ensures that IO sizes that exceed this limit are not polled.
To check EXAScaler file system file layout, run the following command.
$ lfs getstripe <file-path>
Review the output, for example:
lfs getstripe /mnt/ai200/single_stripe/md1.0.0 /mnt/ai200/single_stripe/md1.0.0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 6146 0x1802 0