Troubleshooting and Support
Abstract
This guide helps you with usage issues or bugs that you might encounter using this product.
DIGITS (the Deep Learning GPU Training System) is a webapp for training deep learning models. DIGITS currently supports the TensorFlow framework. DIGITS puts the power of deep learning into the hands of engineers and data scientists.
DIGITS is not a framework. DIGITS is a wrapper for TensorFlow; which provides a graphical web interface to those frameworks rather than dealing with them directly on the command-line.
DIGITS can be used to rapidly train highly accurate deep neural network (DNNs) for image classification, segmentation, object detection tasks, and more. DIGITS simplifies common deep learning tasks such as managing data, designing and training neural networks on multi-GPU systems, monitoring performance in real time with advanced visualizations, and selecting the best performing model from the results browser for deployment. DIGITS is completely interactive so that data scientists can focus on designing and training networks rather than programming and debugging. DIGITS is available through multiple channels such as:
- GitHub download
- NVIDIA’s Docker repository,
nvcr.io
DIGITS uses environment variables for configuration.
Note: Prior to #1091 (up to DIGITS 4.0), DIGITS used configuration files instead of environment variables.
Note:DIGITS is not designed to be run as an exposed external web service.
2.1. Environment variables
This table defines the environment variables DIGITS uses for configuration.
Variable | Example value | Description |
---|---|---|
DIGITS_JOBS_DIR | ~/digits-jobs | Location where job files are stored. Default is $DIGITS_ROOT/digits/jobs. |
DIGITS_LOGFILE_FILENAME | ~/digits.log | File for saving log messages. Default is $DIGITS_ROOT/digits/digits.log. |
DIGITS_LOGFILE_LEVEL | DEBUG | Minimum log message level to be saved (DEBUG/INFO/WARNING/ERROR/CRITICAL). Default is INFO. |
DIGITS_SERVER_NAME | The Big One | The name of the server (accessible in the UI under "Info"). Default is the system hostname. |
DIGITS_MODEL_STORE_URL | http://localhost/modelstore | A list of URL's, separated by comma. Default is the official NVIDIA store. |
DIGITS_URL_PREFIX | /custom-prefix | A path to prepend before every URL. Sets the home-page to be at "http://localhost/custom-prefix" instead of "http://localhost/"/ |
For questions involving the installation or usage of DIGITS see:
- Nvidia DIGITS Installation Guide
- Installing CUDA and the NVIDIA Driver for DIGITS
- Nvidia DIGITS Container Getting Started Guide
- Consult the User Group forum: https://groups.google.com/forum/#!forum/digits-users
3.1. Configuration
If you have another server running on port 80 already, you may need to reconfigure DIGITS to use a different port.
sudo dpkg-reconfigure digits
All other configuration is done with environment variables. See Configuration.md for detailed information about which variables you can change.
- Ubuntu 14.04
- Edit /etc/init/digits.conf
- Add/remove/edit lines that start with env
- Restart with sudo service digits restart
- Ubuntu 16.04
- Edit /lib/systemd/system/digits.service
- Add/remove/edit lines that start with Environment= in the [Service] section
- Restart with sudo systemctl daemon-reload && sudo systemctl restart digits
3.2. Driver installations
If you try to install a new driver while the DIGITS server is running, you'll get an error about CUDA being in use. Shut down the server before installing a driver, and then restart it afterwards.
- Ubuntu 14.04:
sudo service digits stop # (install driver) sudo service digits start
- Ubuntu 16.04:
sudo systemctl stop digits # (install driver) sudo systemctl start digits
3.3. Permissions
The DIGITS server runs as www-data, so keep in mind that prebuilt LMDB datasets used for generic models need to be readable by the www-data user. In particular, the entire chain of directories from / to your data must be readable by www-data.
3.4. Other
If you run into an issue not addressed here, try searching through the GitHub issues and/or the user group.
To file a new issue go to: https://github.com/login?return_to=https%3A%2F%2Fgithub.com%2FNVIDIA%2FDIGITS%2Fissues%2Fnew
To contribute by opening a pull request: https://help.github.com/articles/about-pull-requests/
Note:
You will need to send a signed copy of the Contributor License Agreement to digits@nvidia.com before your change can be accepted.
For the latest Release Notes, see the DIGITS Release Notes Documentation website (http://docs.nvidia.com/deeplearning/digits/digits-release-notes/index.html ). For more information about DIGITS, see:
- DIGITS website (https://developer.nvidia.com/digits )
- DIGITS project (https://github.com/NVIDIA/DIGITS/blob/digits-5.0/README.md )
- GitHub documentation (https://github.com/NVIDIA/nvidia-docker/wiki/DIGITS )
Note: There may be slight variations between the NVIDIA-docker images and this image.
Notice
This document is provided for information purposes only and shall not be regarded as a warranty of a certain functionality, condition, or quality of a product. NVIDIA Corporation (“NVIDIA”) makes no representations or warranties, expressed or implied, as to the accuracy or completeness of the information contained in this document and assumes no responsibility for any errors contained herein. NVIDIA shall have no liability for the consequences or use of such information or for any infringement of patents or other rights of third parties that may result from its use. This document is not a commitment to develop, release, or deliver any Material (defined below), code, or functionality.
NVIDIA reserves the right to make corrections, modifications, enhancements, improvements, and any other changes to this document, at any time without notice.
Customer should obtain the latest relevant information before placing orders and should verify that such information is current and complete.
NVIDIA products are sold subject to the NVIDIA standard terms and conditions of sale supplied at the time of order acknowledgement, unless otherwise agreed in an individual sales agreement signed by authorized representatives of NVIDIA and customer (“Terms of Sale”). NVIDIA hereby expressly objects to applying any customer general terms and conditions with regards to the purchase of the NVIDIA product referenced in this document. No contractual obligations are formed either directly or indirectly by this document.
NVIDIA products are not designed, authorized, or warranted to be suitable for use in medical, military, aircraft, space, or life support equipment, nor in applications where failure or malfunction of the NVIDIA product can reasonably be expected to result in personal injury, death, or property or environmental damage. NVIDIA accepts no liability for inclusion and/or use of NVIDIA products in such equipment or applications and therefore such inclusion and/or use is at customer’s own risk.
NVIDIA makes no representation or warranty that products based on this document will be suitable for any specified use. Testing of all parameters of each product is not necessarily performed by NVIDIA. It is customer’s sole responsibility to evaluate and determine the applicability of any information contained in this document, ensure the product is suitable and fit for the application planned by customer, and perform the necessary testing for the application in order to avoid a default of the application or the product. Weaknesses in customer’s product designs may affect the quality and reliability of the NVIDIA product and may result in additional or different conditions and/or requirements beyond those contained in this document. NVIDIA accepts no liability related to any default, damage, costs, or problem which may be based on or attributable to: (i) the use of the NVIDIA product in any manner that is contrary to this document or (ii) customer product designs.
No license, either expressed or implied, is granted under any NVIDIA patent right, copyright, or other NVIDIA intellectual property right under this document. Information published by NVIDIA regarding third-party products or services does not constitute a license from NVIDIA to use such products or services or a warranty or endorsement thereof. Use of such information may require a license from a third party under the patents or other intellectual property rights of the third party, or a license from NVIDIA under the patents or other intellectual property rights of NVIDIA.
Reproduction of information in this document is permissible only if approved in advance by NVIDIA in writing, reproduced without alteration and in full compliance with all applicable export laws and regulations, and accompanied by all associated conditions, limitations, and notices.
THIS DOCUMENT AND ALL NVIDIA DESIGN SPECIFICATIONS, REFERENCE BOARDS, FILES, DRAWINGS, DIAGNOSTICS, LISTS, AND OTHER DOCUMENTS (TOGETHER AND SEPARATELY, “MATERIALS”) ARE BEING PROVIDED “AS IS.” NVIDIA MAKES NO WARRANTIES, EXPRESSED, IMPLIED, STATUTORY, OR OTHERWISE WITH RESPECT TO THE MATERIALS, AND EXPRESSLY DISCLAIMS ALL IMPLIED WARRANTIES OF NONINFRINGEMENT, MERCHANTABILITY, AND FITNESS FOR A PARTICULAR PURPOSE. TO THE EXTENT NOT PROHIBITED BY LAW, IN NO EVENT WILL NVIDIA BE LIABLE FOR ANY DAMAGES, INCLUDING WITHOUT LIMITATION ANY DIRECT, INDIRECT, SPECIAL, INCIDENTAL, PUNITIVE, OR CONSEQUENTIAL DAMAGES, HOWEVER CAUSED AND REGARDLESS OF THE THEORY OF LIABILITY, ARISING OUT OF ANY USE OF THIS DOCUMENT, EVEN IF NVIDIA HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. Notwithstanding any damages that customer might incur for any reason whatsoever, NVIDIA’s aggregate and cumulative liability towards customer for the products described herein shall be limited in accordance with the Terms of Sale for the product.
VESA DisplayPort
DisplayPort and DisplayPort Compliance Logo, DisplayPort Compliance Logo for Dual-mode Sources, and DisplayPort Compliance Logo for Active Cables are trademarks owned by the Video Electronics Standards Association in the United States and other countries.
HDMI
HDMI, the HDMI logo, and High-Definition Multimedia Interface are trademarks or registered trademarks of HDMI Licensing LLC.
OpenCL
OpenCL is a trademark of Apple Inc. used under license to the Khronos Group Inc.
Trademarks
NVIDIA, the NVIDIA logo, and cuBLAS, CUDA, cuDNN, DALI, DIGITS, DGX, DGX-1, DGX-2, DGX Station, DLProf, Jetson, Kepler, Maxwell, NCCL, Nsight Compute, Nsight Systems, NvCaffe, PerfWorks, Pascal, SDK Manager, Tegra, TensorRT, Triton Inference Server, Tesla, TF-TRT, and Volta are trademarks and/or registered trademarks of NVIDIA Corporation in the U.S. and other countries. Other company and product names may be trademarks of the respective companies with which they are associated.