Contents

• 1. Getting Started
  • 1.1. Overview
  • 1.2. Creating an Example
  • 1.3. Invoking the Command-level NVIDIA HPC Compilers
    • 1.3.1. Command-line Syntax
    • 1.3.2. Command-line Options
  • 1.4. Filename Conventions
    • 1.4.1. Input Files
    • 1.4.2. Output Files
  • 1.5. Fortran, C++ and C Data Types
  • 1.6. Platform-specific considerations
    • 1.6.1. Using the NVIDIA HPC Compilers on Linux
  • 1.7. Site-Specific Customization of the Compilers
    • 1.7.1. Use siterc Files
    • 1.7.2. Using User rc Files
  • 1.8. Common Development Tasks
• 2. Use Command-line Options
  • 2.1. Command-line Option Overview
    • 2.1.1. Command-line Options Syntax
    • 2.1.2. Command-line Suboptions
    • 2.1.3. Command-line Conflicting Options
  • 2.2. Help with Command-line Options
  • 2.3. Getting Started with Performance
    • 2.3.1. Using -fast
    • 2.3.2. Other Performance-Related Options
  • 2.4. Frequently-used Options
  • 2.5. Floating-point Subnormal
• 3. Multicore CPU Optimization
  • 3.1. Overview of Optimization
    • 3.1.1. Local Optimization
    • 3.1.2. Global Optimization
    • 3.1.3. Loop Optimization: Unrolling, Vectorization and Parallelization
    • 3.1.4. Interprocedural Analysis (IPA) and Optimization
    • 3.1.5. Function Inlining
  • 3.2. Getting Started with Optimization
    • 3.2.1. -help
    • 3.2.2. -Minfo
    • 3.2.3. -Mneginfo
    • 3.2.4. -dryrun
    • 3.2.5. -v
  • 3.3. Local and Global Optimization
    • 3.3.1. -Msafeptr
    • 3.3.2. -O
  • 3.4. Loop Unrolling using -Munroll
  • 3.5. Vectorization using -Mvect
    • 3.5.1. Vectorization Sub-options
    • 3.5.2. Vectorization Example Using SIMD Instructions
  • 3.6. Interprocedural Analysis and Optimization using -Mipa
    • 3.6.1. Building a Program Without IPA – Single Step
    • 3.6.2. Building a Program Without IPA – Several Steps
    • 3.6.3. Building a Program Without IPA Using Make
    • 3.6.4. Building a Program with IPA
    • 3.6.5. Building a Program with IPA – Single Step
    • 3.6.6. Building a Program with IPA – Several Steps
    • 3.6.7. Building a Program with IPA Using Make
    • 3.6.8. Questions about IPA
• 4. Using Function Inlining
  • 4.1. Automatic function inlining in C++ and C
  • 4.2. Invoking Procedure Inlining
  • 4.3. Using an Inline Library
  • 4.4. Creating an Inline Library
    • 4.4.1. Working with Inline Libraries
    • 4.4.2. Dependencies
    • 4.4.3. Updating Inline Libraries – Makefiles
  • 4.5. Error Detection during Inlining
  • 4.6. Examples
  • 4.7. Restrictions on Inlining
• 5. Using GPUs
  • 5.1. Overview
  • 5.2. Terminology
  • 5.3. Execution Model
    • 5.3.1. Host Functions
  • 5.4. Memory Model
    • 5.4.1. Separate Host and Accelerator Memory Considerations
      • 5.4.1.1. Accelerator Memory
      • 5.4.1.2. Staging Memory Buffer
      • 5.4.1.3. Cache Management
      • 5.4.1.4. Environment Variables Controlling Device Memory Management
    • 5.4.2. Managed and Unified Memory Modes
      • 5.4.2.1. Managed Memory Mode
      • 5.4.2.2. Unified Memory Mode
    • 5.4.3. Memory Pool Allocator
    • 5.4.4. Interception of Deallocations
    • 5.4.5. Command-line Options Selecting Compiler Memory Modes
  • 5.5. Fortran pointers in device code
  • 5.6. Calling routines in a compute kernel
  • 5.7. Supported Processors and GPUs
  • 5.8. CUDA Versions
  • 5.9. Compute Capability
  • 5.10. PTX JIT Compilation
• 6. Using OpenACC
  • 6.1. OpenACC Programming Model
    • 6.1.1. Levels of Parallelism
    • 6.1.2. Enable OpenACC Directives
    • 6.1.3. OpenACC Support
    • 6.1.4. OpenACC Extensions
  • 6.2. Compiling an OpenACC Program
    • 6.2.1. -[no]acc
    • 6.2.2. -gpu
  • 6.3. OpenACC for Multicore CPUs
  • 6.4. OpenACC with CUDA Unified Memory
  • 6.5. OpenACC Error Handling
  • 6.6. OpenACC and CUDA Graphs
  • 6.7. Host and Device Trip Count Options
    • 6.7.1. When to Use -gpu=tripcount:device or -gpu=tripcount:host
  • 6.8. Environment Variables
  • 6.9. Profiling Accelerator Kernels
  • 6.10. OpenACC Runtime Libraries
    • 6.10.1. Runtime Library Definitions
    • 6.10.2. Runtime Library Routines
  • 6.11. Supported Intrinsics
    • 6.11.1. Supported Fortran Intrinsics Summary Table
    • 6.11.2. Supported C Intrinsics Summary Table
• 7. Using OpenMP
  • 7.1. Environment Variables
  • 7.2. Fallback Mode
  • 7.3. Loop
  • 7.4. OpenMP Subset
  • 7.5. Using metadirective
  • 7.6. Mapping target constructs to CUDA streams
  • 7.7. Noncontiguous Array Sections
  • 7.8. OpenMP with CUDA Unified Memory
  • 7.9. Multiple Device Support
  • 7.10. Interoperability with CUDA
  • 7.11. Interoperability with Other OpenMP Compilers
  • 7.12. GNU STL
• 8. Using Stdpar
  • 8.1. GPU Memory Modes
  • 8.2. Stdpar C++
    • 8.2.1. Introduction to Stdpar C++
    • 8.2.2. NVC++ Compiler Parallel Algorithms Support
      • 8.2.2.1. Enabling Parallel Algorithms with the -stdpar Option
    • 8.2.3. Stdpar C++ Simple Example
    • 8.2.4. OpenACC Implementation of Parallel Algorithms
    • 8.2.5. Coding Guidelines for GPU-accelerating Parallel Algorithms
      • 8.2.5.1. Parallel Algorithms and Device Function Annotations
      • 8.2.5.2. Data Management in Parallel Algorithms
      • 8.2.5.3. Parallel Algorithms and Function Pointers
      • 8.2.5.4. Random Access Iterators
      • 8.2.5.5. Interoperability with the C++ Standard Library
      • 8.2.5.6. No Exceptions in GPU Code
    • 8.2.6. NVC++ Experimental Features
      • 8.2.6.1. Multi-dimensional Spans
      • 8.2.6.2. Senders and Receivers
      • 8.2.6.3. Linear Algebra
    • 8.2.7. Stdpar C++ Larger Example: LULESH
    • 8.2.8. Interoperability with OpenACC
      • 8.2.8.1. Data Management Directives
      • 8.2.8.2. External Device Function Annotations
    • 8.2.9. Getting Started with Parallel Algorithms for GPUs
      • 8.2.9.1. Supported NVIDIA GPUs
      • 8.2.9.2. Supported CUDA Versions
  • 8.3. Stdpar Fortran
    • 8.3.1. Calling Routines in DO CONCURRENT on the GPU
    • 8.3.2. GPU Data Management
    • 8.3.3. Interoperability with OpenACC
    • 8.3.4. Interoperability with CUDA Fortran
• 9. PCAST
  • 9.1. Overview
  • 9.2. PCAST with a “Golden” File
  • 9.3. PCAST with OpenACC
  • 9.4. Limitations
  • 9.5. Environment Variables
• 10. Using MPI
  • 10.1. Using Open MPI on Linux
  • 10.2. Using MPI Compiler Wrappers
  • 10.3. Testing and Benchmarking
• 11. Creating and Using Libraries
  • 11.1. Using builtin Math Functions in C++ and C
  • 11.2. Using System Library Routines
  • 11.3. Creating and Using Shared Object Files on Linux
    • 11.3.1. Procedure to create a use a shared object file
    • 11.3.2. ldd Command
  • 11.4. Using LIB3F
  • 11.5. LAPACK, BLAS and FFTs
  • 11.6. Linking with ScaLAPACK
  • 11.7. The C++ Standard Template Library
  • 11.8. NVIDIA Performance Libraries (NVPL)
  • 11.9. Linking with the nvmalloc Library
    • 11.9.1. Using nvmalloc with Shared Libraries
• 12. Environment Variables
  • 12.1. Setting Environment Variables
    • 12.1.1. Setting Environment Variables on Linux
  • 12.2. HPC Compiler Related Environment Variables
  • 12.3. HPC Compilers Environment Variables
    • 12.3.1. FORTRANOPT
    • 12.3.2. FORT_FMT_RECL
    • 12.3.3. GMON_OUT_PREFIX
    • 12.3.4. LD_LIBRARY_PATH
    • 12.3.5. MANPATH
    • 12.3.6. NO_STOP_MESSAGE
    • 12.3.7. PATH
    • 12.3.8. NVCOMPILER_FPU_STATE
    • 12.3.9. NVCOMPILER_TERM
    • 12.3.10. NVCOMPILER_TERM_DEBUG
    • 12.3.11. PWD
    • 12.3.12. STATIC_RANDOM_SEED
    • 12.3.13. TMP
    • 12.3.14. TMPDIR
  • 12.4. Using Environment Modules on Linux
  • 12.5. Stack Traceback and JIT Debugging
• 13. Distributing Files - Deployment
  • 13.1. Deploying Applications on Linux
    • 13.1.1. Runtime Library Considerations
    • 13.1.2. 64-bit Linux Considerations
    • 13.1.3. Linux Redistributable Files
    • 13.1.4. Restrictions on Linux Portability
    • 13.1.5. Licensing for Redistributable (REDIST) Files
• 14. Inter-language Calling
  • 14.1. Overview of Calling Conventions
  • 14.2. Inter-language Calling Considerations
  • 14.3. Functions and Subroutines
  • 14.4. Upper and Lower Case Conventions, Underscores
  • 14.5. Compatible Data Types
    • 14.5.1. Fortran Named Common Blocks
  • 14.6. Argument Passing and Return Values
    • 14.6.1. Passing by Value (%VAL)
    • 14.6.2. Character Return Values
    • 14.6.3. Complex Return Values
  • 14.7. Array Indices
  • 14.8. Examples
    • 14.8.1. Example – Fortran Calling C
    • 14.8.2. Example - C Calling Fortran
    • 14.8.3. Example – C++ Calling C
    • 14.8.4. Example – C Calling C ++
    • 14.8.5. Example – Fortran Calling C++
    • 14.8.6. Example – C++ Calling Fortran
• 15. Programming Considerations for 64-Bit Environments
  • 15.1. Data Types in the 64-Bit Environment
    • 15.1.1. C++ and C Data Types
    • 15.1.2. Fortran Data Types
  • 15.2. Large Static Data in Linux
  • 15.3. Large Dynamically Allocated Data
  • 15.4. 64-Bit Array Indexing
  • 15.5. Compiler Options for 64-bit Programming
  • 15.6. Practical Limitations of Large Array Programming
  • 15.7. Medium Memory Model and Large Array in C
  • 15.8. Medium Memory Model and Large Array in Fortran
  • 15.9. Large Array and Small Memory Model in Fortran
• 16. C++ and C Inline Assembly and Intrinsics
  • 16.1. Inline Assembly
  • 16.2. Extended Inline Assembly
    • 16.2.1. Output Operands
    • 16.2.2. Input Operands
    • 16.2.3. Clobber List
    • 16.2.4. Additional Constraints
    • 16.2.5. Simple Constraints
    • 16.2.6. Machine Constraints
    • 16.2.7. Multiple Alternative Constraints
    • 16.2.8. Constraint Modifiers
  • 16.3. Operand Aliases
  • 16.4. Assembly String Modifiers
  • 16.5. Extended Asm Macros
  • 16.6. Intrinsics