Visualizing Training
Visualization is a very important part of training a Deep Neural Network (DNN). Training an DNN involves designing complex neural networks with models having parameters to the tune of millions, and iterating over large datasets. Therefore, understanding how this training progresses over time, visualizing structure of the model graph, and what are the statistics of the model weights are of significant importance.
TAO 3.22.05 introduces integration of the following computer vision networks with TensorBoard.
DetectNet-v2
FasterRCNN
Image Classification
MultiTask Classification
RetinaNet
YOLOv4/YOLOv4-Tiny
YOLOv3
MaskRCNN
UNet
SSD
DSSD
The networks supported in TAO supports visualizing
Scalar plots such as training loss, validation loss and learning rate
Histograms for weights
Images
In order to enable tensorboard while training, you can simply add the following spec element to
the training_config
element of the configuration/experiment spec file.
visualizer{
enabled: true
}
For detailed information about the configurable elements of the visualizer, please review the table corresponding
to the training_config
element of the network, in the Creating an Experiment Spec file
of the respective
network.
Installing tensorboard
Installing tensorboard is as simple as running a simple
pip
installation command.python -m pip install tensorboard
Invoking Tensorboard
Once you have installed tensorboard in your python environment, you may instantiate a tensorboard session by running the following command.
tensorboard --logdir $RESULTS_DIR --host 0.0.0.0 --port 8080
where
$RESULTS_DIR
is the path to where theevents.out.tfevents.*
files are stored from the training experiment.
If you would like to visualize results from multiple experiments side-by-side in a single tensorboard session, you may do so by adding multiple directories to the same tensorboard path, as shown in the command below.
tensorboard --logdir_spec experiment_name_1:${RESULTS_DIR_1},experiment_name_2:${RESULTS_DIR_2} \
--host 0.0.0.0 --port 8080
For more information about the TensorBoard client, please refer to the official documentation. You may also refer to the getting started documentation and FAQ.