Contents

White paper covering the most common issues related to NVIDIA GPUs.

• 1. Preface
  • 1.1. What Is This Document?
  • 1.2. Who Should Read This Guide?
  • 1.3. Assess, Parallelize, Optimize, Deploy
    • 1.3.1. Assess
    • 1.3.2. Parallelize
    • 1.3.3. Optimize
    • 1.3.4. Deploy
  • 1.4. Recommendations and Best Practices
  • 1.5. Assessing Your Application
• 2. Heterogeneous Computing
  • 2.1. Differences between Host and Device
  • 2.2. What Runs on a CUDA-Enabled Device?
• 3. Application Profiling
  • 3.1. Profile
    • 3.1.1. Creating the Profile
    • 3.1.2. Identifying Hotspots
    • 3.1.3. Understanding Scaling
      • 3.1.3.1. Strong Scaling and Amdahl’s Law
      • 3.1.3.2. Weak Scaling and Gustafson’s Law
      • 3.1.3.3. Applying Strong and Weak Scaling
• 4. Parallelizing Your Application
• 5. Getting Started
  • 5.1. Parallel Libraries
  • 5.2. Parallelizing Compilers
  • 5.3. Coding to Expose Parallelism
• 6. Getting the Right Answer
  • 6.1. Verification
    • 6.1.1. Reference Comparison
    • 6.1.2. Unit Testing
  • 6.2. Debugging
  • 6.3. Numerical Accuracy and Precision
    • 6.3.1. Single vs. Double Precision
    • 6.3.2. Floating Point Math Is not Associative
    • 6.3.3. IEEE 754 Compliance
    • 6.3.4. x86 80-bit Computations
• 7. Optimizing CUDA Applications
• 8. Performance Metrics
  • 8.1. Timing
    • 8.1.1. Using CPU Timers
    • 8.1.2. Using CUDA GPU Timers
  • 8.2. Bandwidth
    • 8.2.1. Theoretical Bandwidth Calculation
    • 8.2.2. Effective Bandwidth Calculation
    • 8.2.3. Throughput Reported by Visual Profiler
• 9. Memory Optimizations
  • 9.1. Data Transfer Between Host and Device
    • 9.1.1. Pinned Memory
    • 9.1.2. Asynchronous and Overlapping Transfers with Computation
    • 9.1.3. Zero Copy
    • 9.1.4. Unified Virtual Addressing
  • 9.2. Device Memory Spaces
    • 9.2.1. Coalesced Access to Global Memory
      • 9.2.1.1. A Simple Access Pattern
      • 9.2.1.2. A Sequential but Misaligned Access Pattern
      • 9.2.1.3. Effects of Misaligned Accesses
      • 9.2.1.4. Strided Accesses
    • 9.2.2. L2 Cache
      • 9.2.2.1. L2 Cache Access Window
      • 9.2.2.2. Tuning the Access Window Hit-Ratio
    • 9.2.3. Shared Memory
      • 9.2.3.1. Shared Memory and Memory Banks
      • 9.2.3.2. Shared Memory in Matrix Multiplication (C=AB)
      • 9.2.3.3. Shared Memory in Matrix Multiplication (C=AAT)
      • 9.2.3.4. Asynchronous Copy from Global Memory to Shared Memory
    • 9.2.4. Local Memory
    • 9.2.5. Texture Memory
      • 9.2.5.1. Additional Texture Capabilities
    • 9.2.6. Constant Memory
    • 9.2.7. Registers
      • 9.2.7.1. Register Pressure
  • 9.3. Allocation
  • 9.4. NUMA Best Practices
• 10. Execution Configuration Optimizations
  • 10.1. Occupancy
    • 10.1.1. Calculating Occupancy
  • 10.2. Hiding Register Dependencies
  • 10.3. Thread and Block Heuristics
  • 10.4. Effects of Shared Memory
  • 10.5. Concurrent Kernel Execution
  • 10.6. Multiple contexts
• 11. Instruction Optimization
  • 11.1. Arithmetic Instructions
    • 11.1.1. Division Modulo Operations
    • 11.1.2. Loop Counters Signed vs. Unsigned
    • 11.1.3. Reciprocal Square Root
    • 11.1.4. Other Arithmetic Instructions
    • 11.1.5. Exponentiation With Small Fractional Arguments
    • 11.1.6. Math Libraries
    • 11.1.7. Precision-related Compiler Flags
  • 11.2. Memory Instructions
• 12. Control Flow
  • 12.1. Branching and Divergence
  • 12.2. Branch Predication
• 13. Deploying CUDA Applications
• 14. Understanding the Programming Environment
  • 14.1. CUDA Compute Capability
  • 14.2. Additional Hardware Data
  • 14.3. Which Compute Capability Target
  • 14.4. CUDA Runtime
• 15. CUDA Compatibility Developer’s Guide
  • 15.1. CUDA Toolkit Versioning
  • 15.2. Source Compatibility
  • 15.3. Binary Compatibility
    • 15.3.1. CUDA Binary (cubin) Compatibility
  • 15.4. CUDA Compatibility Across Minor Releases
    • 15.4.1. Existing CUDA Applications within Minor Versions of CUDA
      • 15.4.1.1. Handling New CUDA Features and Driver APIs
      • 15.4.1.2. Using PTX
      • 15.4.1.3. Dynamic Code Generation
      • 15.4.1.4. Recommendations for building a minor-version compatible library
      • 15.4.1.5. Recommendations for taking advantage of minor version compatibility in your application
• 16. Preparing for Deployment
  • 16.1. Testing for CUDA Availability
  • 16.2. Error Handling
  • 16.3. Building for Maximum Compatibility
  • 16.4. Distributing the CUDA Runtime and Libraries
    • 16.4.1. CUDA Toolkit Library Redistribution
      • 16.4.1.1. Which Files to Redistribute
      • 16.4.1.2. Where to Install Redistributed CUDA Libraries
• 17. Deployment Infrastructure Tools
  • 17.1. Nvidia-SMI
    • 17.1.1. Queryable state
    • 17.1.2. Modifiable state
  • 17.2. NVML
  • 17.3. Cluster Management Tools
  • 17.4. Compiler JIT Cache Management Tools
  • 17.5. CUDA_VISIBLE_DEVICES
• 18. Recommendations and Best Practices
  • 18.1. Overall Performance Optimization Strategies
• 19. nvcc Compiler Switches
  • 19.1. nvcc
• 20. Notices
  • 20.1. Notice
  • 20.2. OpenCL
  • 20.3. Trademarks