The various options for NVVS are as follows:

To display the list of GPUs available on the system.

user@hostname
$ nvvs -g
			
NVIDIA Validation Suite (version 352.00)

Supported GPUs available:
	[0000:01:00.0] -- Tesla K40c
	[0000:05:00.0] -- Tesla K20c
	[0000:06:00.0] -- Tesla K20c
		

An example "quick" test (explained later) using a custom configuration file.

user@hostname
$ nvvs -c Tesla_K40c_quick.conf
			
NVIDIA Validation Suite (version 352.00)

	Software
		Blacklist ......................................... PASS
		NVML Library ...................................... PASS
		CUDA Main Library ................................. PASS
		CUDA Toolkit Libraries ............................ PASS
		Permissions and OS-related Blocks ................. PASS
		Persistence Mode .................................. PASS
		Environmental Variables ........................... PASS			
		

To output an encrypted debug file at the highest debug level to send to NVIDIA for analysis after a problem.

user@hostname
$ nvvs -c Tesla_K40c_medium.conf -d 5 -l debug.log

NVIDIA Validation Suite (version 352.00)

	Software
		Blacklist ......................................... PASS
		NVML Library ...................................... PASS
		CUDA Main Library ................................. PASS
		CUDA Toolkit Libraries ............................ PASS
		Permissions and OS-related Blocks ................. PASS
		Persistence Mode .................................. PASS
		Environmental Variables ........................... PASS
	Hardware
		Memory GPU0 ....................................... PASS
	Integration
		PCIe .............................................. FAIL
           *** GPU 0 is running at PCI link width 8X, which is below the minimum allowed link width of 16X (parameter: 
min_pci_width)"			
		

The output file, debug.log would then be returned to NVIDIA.

The NVVS configuration file is a YAML-formatted (e.g. human-readable JSON) text file with three main stanzas controlling the various tests and their execution.

The general format of a configuration file consists of:

%YAML 1.2
---
      
globals:
  key1: value
  key2: value
      
test_suite_name:
- test_class_name1:
    test_name1:
      key1: value
      key2: value
      subtests:
        subtest_name1:
          key1: value
          key2: value
    test_name2:
      key1: value
      key2: value
-test_class_name2:
    test_name3:
      key1: value
      key2: value
      
gpus:
- gpuset: name
  properties:
    key1: value
    key2: value
  tests:
    name: test_suite_name
    

There are three distinct sections: globals, test_suite_name, and gpus each with its own subsection of parameters and as is with any YAML document, indentation is important thus if errors are generated from your own configuration files please refer to this example for indentation reference.

The gpus stanza may consist of one or more gpusets which will each match zero or more GPUs on the system based on their properties(a match of zero will produce an error).

The matching rules are based off of exclusion. First, the list of supported GPUs is taken and if no properties tag is given then all GPUs will be used in the test. Because a UUID or PCIe Bus ID can only match a single GPU, if those properties are given then only that GPU will be used if found. The remaining properties, index, brand, and name work in an "AND" fashion such that, if specified, the result must match at least one GPU on the system for a test to be performed.

For example, if name is set to "Tesla K40c" and index is set to "0" NVVS will error if index 0 is not a Tesla K40c. By specifying both brand and index a user may limit a test to specific "Tesla" cards for example. In this version of NVVS, all matching GPUs must be homogeneous.

The second identifier for a gpuset is tests. This parameter specifies either the suite of tests that a user wishes to run or the test itself.

At present the following suites are available:

An individual test can also be specified. Currently the keywords are: Memory, Diagnostic, Targeted Stress, Targeted Power, PCIe, SM Stress, and Memory Bandwidth. Please see the "custom" section in the next subchapter to configure and tweak the parameters when this method is used.

The format of the NVVS configuration file is designed for extensibility. Each test suite above can be customized in a number of ways described in detail in the following chapter for each test. Individual tests belong to a specific class of functionality which, when wanting to customize specific parameters, must also be specified.

The classes and the respective tests they perform are as follows:

Some tests also have subtests that can be enabled by using the subtests keyword and then hierarchically adding the subtest parameters desired beneath. An example would be the PCIe Bandwidth test which may have a section that looks similar to this:

long:
- integration:
    pcie:
      test_unpinned: false
      subtests:
        h2d_d2h_single_pinned:
          min_bandwidth: 20
          min_pci_width: 16
        

When only a specific test is given in the GPU set portion of the configuration file, both the suite and class of the test are custom. For example:

%YAML 1.2 
---

globals:
  logfile: nvvs.log

custom:
- custom:
    targeted stress:
      test_duration: 60

gpus:
- gpuset: all_K40c
  properties:
    name: Tesla K40c
  tests: 
    - name: targeted stress
        

The deployment plugin's purpose is to verify the compute environment is ready to run Cuda applications and is able to load the NVML library.

The HW Diagnostic Plugin is designed to identify HW failures on GPU silicon and board-level components, extending out to the PCIE and NVLINK interfaces. It is not intended to identify HW or system level issues beyond the NVIDIA-provided HW. Nor is it intended to identify SW level issues above the HW, e.g. in the NVIDIA driver stack. The plugin runs a series of tests that target GPU computational correctness, GDDR/HBM memory resiliency, GPU and SRAM high power operation, SM stress and NVLINK/PCIE correctness. The plugin can run with several combinations of tests corresponding to medium and long NVVS operational modes. This plugin will take about three minutes to execute.

The plugin produces a simple pass/fail output. A failing output means that a potential HW issue has been found. However, the NVVS HW Diagnostic Plugin is not by itself a justification for GPU RMA. Any failure in the plugin should be followed by execution of the full NVIDIA Field Diagnostic after the machine has been taken offline. Only a failure of the Field Diagnostic tool constitutes grounds for RMA. Since the NVVS HW Diagnostic Plugin is a functional subset of the Field Diagnostic a failure in the plugin is a strong indicator of a future Field Diagnostic failure.

The GPU bandwidth plugin's purpose is to measure the bandwidth and latency to and from the GPUs and the host.

The purpose of the Memory Bandwidth plugin is to validate that the bandwidth of the framebuffer of the GPU is above a preconfigured threshold.

The SM performance plugin’s purpose is to bring the Streaming Multiprocessors (SMs) of the target GPU(s) to a target performance level in gigaflops by doing large matrix multiplications using cublas. Unlike the Targeted Stress plugin, the SM stress plugin does not copy the source arrays to the GPU before every matrix multiplication. This allows the SM performance plugin's performance to not be capped by device to host bandwidth. The plugin calculates how many matrix operations per second are necessary to achieve the configured performance target and fails if it cannot achieve that target.

This plugin should be used to watch for thermal, power and related anomalies while the target GPU(s) are under realistic load conditions. By setting the appropriate parameters a user can ensure that all GPUs in a node or cluster reach desired performance levels. Further analysis of the generated stats can also show variations in the required power, clocks or temperatures to reach these targets, and thus highlight GPUs or nodes that are operating less efficiently.

The Targeted Stress plugin’s purpose is to bring the GPU to a target performance level in gigaflops by doing large matrix multiplications using cublas. The plugin calculates how many matrix operations per second are necessary to achieve the configured performance target and fails if it cannot achieve that target.

This plugin should be used to watch for thermal, power and related anomalies while the target GPU(s) are under realistic load conditions. By setting the appropriate parameters a user can ensure that all GPUs in a node or cluster reach desired performance levels. Further analysis of the generated stats can also show variations in the required power, clocks or temperatures to reach these targets, and thus highlight GPUs or nodes that are operating less efficiently.

The purpose of the power plugin is to bring the GPUs to a preconfigured power level in watts by gradually increasing the compute load on the GPUs until the desired power level is achieved. This verifies that the GPUs can sustain a power level for a reasonable amount of time without problems like thermal violations arising.

The JSON output format is keyed based off of the "stats" keys given in each test overview from Chapter 3. These standard JSON files can be processed in any number of ways but two example Python scripts have been provided to aid in visualization in the default installation directory.. The first is a JSON to comma-separated value script (json2csv.py) which can be used to import key values in to a graphing spreadsheet. Proper usage would be:

user@hostname
$ python json2csv.py -i stats_targeted_performance.json -o stats.csv -k gpu_temperature,power_usage
    

Also provided is an example Python script that uses the pygal library to generate readily viewable scalar vector graphics charts (json2svg.py), able to be opened in any browser. Proper usage would be:

user@hostname
$ python json2svg.py -i stats_targeted_performance.json -o stats.svg -k gpu_temperature,power_usage