Visualizing Training#

Visualization is a very important part of training a Deep Neural Network (DNN). Training an DNN involves designing complex neural networks with models having parameters to the tune of millions, and iterating over large datasets. Therefore, understanding how this training progresses over time, visualizing structure of the model graph, and what are the statistics of the model weights are of significant importance.

TAO networks support training visualization with TensorBoard.

TAO supports visualizing the following:

  1. Scalar plots such as training loss, validation loss and learning rate

  2. Histograms for weights

  3. Images

Enabling TensorBoard During Training#

Set visualizer.enabled: true in your experiment specification. For networks that use the legacy TF1 prototxt specification, add the equivalent visualizer { enabled: true } block to the training_config element.

For the configurable elements of the visualizer, refer to the visualizer table on the network’s Creating an Experiment Specification File page.

Visualizing using TensorBoard#

Installing TensorBoard#

  • Installing TensorBoard is as simple as running a simple pip installation command.

    python -m pip install tensorboard

Invoking TensorBoard#

  • Once you have installed TensorBoard in your python environment, you may instantiate a TensorBoard session by running the following command.

    tensorboard --logdir $RESULTS_DIR --host 0.0.0.0 --port 8080

    where $RESULTS_DIR is the path to where the events.out.tfevents.* files are stored from the training experiment.

Note

If you would like to visualize results from multiple experiments side-by-side in a single TensorBoard session, you may do so by adding multiple directories to the same TensorBoard path, as shown in the command below.

tensorboard --logdir_spec experiment_name_1:${RESULTS_DIR_1},experiment_name_2:${RESULTS_DIR_2} \
            --host 0.0.0.0 --port 8080

Additional Resources#

For more information about the TensorBoard client, please refer to the official documentation. You may also refer to the getting started documentation and FAQ.