Step #2: Visualize Training Using TensorBoard

Use the lab link on the left to open Tensorboard to visualize training of the model at real time on Tensorboard. Visualization is also a very important part of training a Deep Neural Network (DNN). Training a DNN involves designing complex neural networks with models. These have millions parameters which can be tuned by iterating over large datasets. Therefore visualizing the structure of the model graph allows us to understand how the training is progressing over time, and what statistics of the model weights are of significant importance.

The networks supported in TAO Toolkit supports visualizing

  1. Scalar plots such as training loss, validation loss and learning rate

  2. Histograms for weigts

  3. Images

Inorder to enable tensorboard while training, you can simply add the following spec element to the training_config element of the configuration/experiment spec file.

Copy
Copied!
            

visualizer{ enabled: true }

Information on how to set up the Tensorboard for the TAO toolkit outside launchpad can be found here.

© Copyright 2022-2023, NVIDIA. Last updated on Jan 10, 2023.