MXNet Release 19.09

The NVIDIA container image for MXNet, release 19.09, is available on NGC.

Contents of the MXNet container

This container image contains the complete source of the version of MXNet in /opt/mxnet. It is pre-built and installed to the Python path.

The container also includes the following:

Driver Requirements

Release 19.09 is based on NVIDIA CUDA 10.1.243, which requires NVIDIA Driver release 418.xx. However, if you are running on Tesla (for example, T4 or any other Tesla board), you may use NVIDIA driver release 396, 384.111+ or 410. However, if you are running on Tesla (Tesla V100, Tesla P4, Tesla P40, or Tesla P100), you may use NVIDIA driver release 384.111+ or 410. The CUDA driver's compatibility package only supports particular drivers. For a complete list of supported drivers, see the CUDA Application Compatibility topic. For more information, see CUDA Compatibility and Upgrades.

GPU Requirements

Release 19.09 supports CUDA compute capability 6.0 and higher. This corresponds to GPUs in the Pascal, Volta, and Turing families. Specifically, for a list of GPUs that this compute capability corresponds to, see CUDA GPUs. For additional support details, see Deep Learning Frameworks Support Matrix.

Key Features and Enhancements

This MXNet release includes the following key features and enhancements.

Tensor Core Examples

These examples focus on achieving the best performance and convergence from NVIDIA Volta Tensor Cores by using the latest deep learning example networks for training.

Each example model trains with mixed precision Tensor Cores on Volta, therefore you can get results much faster than training without tensor cores. This model is tested against each NGC monthly container release to ensure consistent accuracy and performance over time. This container includes the following tensor core examples.

Automatic Mixed Precision (AMP)

Training deep learning networks is a very computationally intensive task. Novel model architectures tend to have an increasing number of layers and parameters, which slows down training. Fortunately, new generations of training hardware as well as software optimizations make training these new models a feasible task.

Most of the hardware and software training optimization opportunities involve exploiting lower precision like FP16 in order to utilize the Tensor Cores available on new Volta and Turing GPUs. While training in FP16 showed great success in image classification tasks, other more complicated neural networks typically stayed in FP32 due to difficulties in applying the FP16 training guidelines that are needed to ensure proper model training.

That is where AMP (Automatic Mixed Precision) comes into play- it automatically applies the guidelines of FP16 training, using FP16 precision where it provides the most benefit, while conservatively keeping in full FP32 precision operations unsafe to do in FP16.

The MXNet AMP tutorial, located in /opt/mxnet/nvidia-examples/AMP/AMP_tutorial.md inside this container, shows how to get started with mixed precision training using AMP for MXNet, using by example the SSD network from GluonCV.

For more information about AMP, see the Training With Mixed Precision Guide.

Known Issues

  • The MXNet KVStore GPU peer-to-peer communication tree discovery, as of release 18.09, is not compatible with DGX-1V. Only users that set the environment variable MXNET_KVSTORE_USETREE=1 will experience issues, which will be resolved in a subsequent release. Issue tracked under 13341.
  • There is a known performance issue with the resnet50v1.5 MXNet 19.09 script on T4 and Titan RTX GPUs. The NHWC_BATCHNORM_LAUNCH_MARGIN environment variable contains a sub-optimal value. As a workaround, remove line 78 ( os.environ['NHWC_BATCHNORM_LAUNCH_MARGIN'] = "64") from the /opt/mxnet/nvidia-examples/resnet50v1.5/runner file.
  • The default setting of the environment variable MXNET_GPU_COPY_NTHREADS=1 in the container may not be optimal for all networks. Networks with a high ratio of parameters and computation, like AlexNet, may achieve greater multi-GPU training speeds with the setting MXNET_GPU_COPY_NTHREADS=2. Users are encouraged to try this setting for their own use case.
  • There is a known issue in the BERT QA demo, where for some values of sequence length the inference may fail with CUDA Driver: operation failed due to a previous error during captureerror. To test those values of sequence length, change the line export MXNET_EXPERIMENTAL_ENABLE_CUDA_GRAPH=1 to export MXNET_EXPERIMENTAL_ENABLE_CUDA_GRAPH=0 in the test_bert_inference script inside the demo directory.