Deep Learning Profiler 19.06 Release Notes

Description

Deep Learning Profiler (DLProf) is a tool for profiling deep learning models to help data scientists understand and improve performance of their models visually via Tensorboard or by analyzing text reports. It also helps understand resource usage when models are trained.

Key Features

The key features of DLProf v0.3.7/r19.06:
  • Easy Profiling: Just provide graphdef file and prefix training script with “dlprof” to automatically get Tensorboard event files with profile data.
  • Visualization: Modified version of Tensorboard shows profiling data on a familiar interface.
  • Reports: Several profiling reports can be generated to provide maximum value.

Known Issues

  • This is a very early version software. It is only accessible in the NGC TensorFlow container. Currently we do not support profiling models with the XLA option. Some of the timing data in Tensorboard may not add up. The default key node of 'global_step/add' assumed by DLProf may not exist in some models. If no iterations are being detected, users should try running the Deep Learning Profiler with a different key node --key_node={new_key_node}.

    Some models execute nodes in the final iterations that are not representative of a normal training iteration. These final iterations may not contain any GPU activity.

  • DLProf and Nsight Systems in the container will not work with NVIDIA GPU Drivers newer than version 418.

Resolved Issues

There are no resolved issues in this release.