Running PyTorch

This chapter goes through some example runs with PyTorch. Some output messages have been removed or simplified in the runs for readability.

The sample runs assume that PyTorch and its extra libraries have been installed from the NVIDIA Base Command Manager repository with a package manager. For example with:

yum install cm-pytorch-py37-cuda10.2-gcc8 cm-pytorch-extra-py37-cuda10.2-gcc8

In addition to requiring the extra libraries, PyTorch requires an OpenMPI implementation to work. Chapter 3 of the User Manual describes the different OpenMPI packages that the BCM repositories provide. The different OpenMPI packages allow the user to choose which one to use. For example, depending on which interconnect is being used, or depending on if CUDA support is required.

In this chapter, the cm-openmpi4-cuda10.2-ofed50-gcc8 package is used.

More information on the examples can be found at https://github.com/pytorch/examples.

3.1 Variational Autoencoders

The following example shows how to train Variational Autoencoders (original paper: “Auto-Encoding Variational Bayes” by Diederik P Kingma and Max Welling, https://arxiv.org/abs/1312.6114), powerful generative models that can be used for a wide variety of applications.

The example code is included in the PyTorch extra (cm-pytorch-extra-*) package. Once its module has been loaded, the example directory is defined with the CM_PYTORCH_EXTRA environment variable.

Example

[root@basecm10 ~]# module load shared
[root@basecm10 ~]# module load pytorch-extra-py37-cuda10.2-gcc8
[root@basecm10 ~]# module load openmpi4-cuda10.2-ofed50-gcc8
[root@basecm10 ~]# cd ${CM_PYTORCH_EXTRA}/vae

The Variational Autoencoders network is trained by default for 10 epochs. The required dataset can automatically be downloaded and extracted with:

Example

[root@basecm10 vae]# python3.7 main.py
Downloading http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz to
../data/MNIST/raw/train-images-idx3-ubyte.gz
9920512it [00:04, 2041116.37it/s]
Extracting ../data/MNIST/raw/train-images-idx3-ubyte.gz to ../data/MNIST/raw
...
Processing...
Done!
Train Epoch: 1 [0/60000 (0%)]   Loss: 550.187805
Train Epoch: 1 [1280/60000 (2%)]    Loss: 323.104736
Train Epoch: 1 [2560/60000 (4%)]    Loss: 237.460938
...
Train Epoch: 1 [58880/60000 (98%)]  Loss: 130.540909
====> Epoch: 1 Average loss: 164.1742
====> Test set loss: 127.8219
Train Epoch: 2 [0/60000 (0%)]   Loss: 127.949753
...
Train Epoch: 10 [58880/60000 (98%)] Loss: 107.980888
====> Epoch: 10 Average loss: 106.1472
====> Test set loss: 105.8715

The output sampled digits can be found in the results directory:

[root@basecm10 vae]# ls results/
reconstruction_10.png  reconstruction_4.png  reconstruction_8.png  sample_2.png  sample_6.png
reconstruction_1.png   reconstruction_5.png  reconstruction_9.png  sample_3.png  sample_7.png
reconstruction_2.png   reconstruction_6.png  sample_10.png         sample_4.png  sample_8.png
reconstruction_3.png   reconstruction_7.png  sample_1.png          sample_5.png  sample_9.png
[root@basecm10 vae]#