Optimizing VPU Programs#

For this tutorial, we are estimating the ideal performance (Speed of Light, SOL) of the convolution vector code, and showing optimization techniques to help reach SOL. This tutorial is broken down into the following parts:

  • Determining the Speed-of-Light (SOL) performance of the convolution loop. You are introduced to the VPU’s rules for operation slotting in its Very Long Instruction Word (VLIW) execution packets.

  • Demonstrate how to examine device code disassembly of the vector convolution code and see how the compiler schedule compares against the SOL calculation.

  • Show how to apply loop unrolling and software pipelining to improve loop scheduling.

The techniques apply to both NVVPU and ASIP Programmer. Both compilers accept VPU syntax such as chess_unroll_loop and produce similar packet-oriented disassembly, although exact instruction selection and scheduling can differ.

Optimization Playbook#

Before diving into the convolution case study, it is worth establishing a repeatable framework for optimizing PVA code. New users frequently jump straight to loop pragmas; in practice, the largest gains come from picking the right remedy for the bottleneck you actually have. A reliable workflow is:

  1. Measure first. Capture a baseline with Nsight Systems and the on-chip perfmon counters described in Profiling Tools. Record both the steady-state cycles-per-tile in VPU and the DMA throughput / VPU utilization values reported under SOC Metrics.

  2. Classify the bottleneck. Compare the measured cycles-per-pixel to your SOL estimate. Use the table below to identify which of the three regimes you are in.

    Regime

    Symptom

    Where to look next

    Compute-bound

    VPU utilization near 100%; cycles-per-tile close to SOL; DMA visibly idle.

    The remainder of this tutorial: VLIW packing, chess_unroll_loop and chess_prepare_for_pipelining pragmas, vector intrinsic selection, eliminating scalar hazards. Also consider offloading work off the VPU to PVA’s other compute engines, which run asynchronously: the Sampler API for table lookups and bilinear interpolation (DLUT), and on GEN3 the PVA Accelerated Primitives Library which dispatches fixed-function PPE kernels.

    Memory-bound

    DMA channel waits dominate the timeline; VPU stalls waiting on tile sync.

    The general remedy is to maximize the overlap of DMA transfers and VPU computation. Reach for Raster DataFlow Double Buffering and VMEM Buffers and DataFlows, configuring Raster/Tensor DataFlows with DOUBLE or CIRCULAR buffers, or implement manual double-buffering using cupva::SequenceDataFlow. In the VPU code, trigger the DMA for the next tile as early as possible and sync as late as possible before consuming a buffer. If the VPU still does not have enough work to keep it busy while DMA transfers the next buffer, fuse adjacent operations into the same CmdProgram so each task carries more meaningful work.

    I/O-bound (DRAM bandwidth)

    SOC Metrics row shows DMA throughput at the platform’s DRAM ceiling.

    The first lever is to reduce redundant DRAM traffic — fold passes together so the same buffer is read once, and use L2 SRAM to cache buffers that the same task re-reads (see “When is L2 SRAM not the right choice?” at the end of that tutorial for when L2 does not help). Improving tile shape to keep tiles within VMEM and reduce halo over-read often shifts the workload out of this regime entirely.

  3. Apply one remedy at a time. Re-measure after each change. Most performance regressions in PVA code come from compounding two changes whose interaction was not measured.

  4. Re-evaluate SOL. Optimization that succeeds often moves you into a new regime (for example, fixing a memory stall now exposes a compute ceiling). Recompute your SOL target against the new bottleneck and continue.

The remainder of this tutorial works through step 3 in detail for a compute-bound convolution kernel. The same playbook applies to the other regimes; the cross-referenced tutorials above are organized to follow the same measure → classify → apply pattern.

Computing the Speed-of-Light (SOL) Performance#

In this tutorial, we are optimizing the performance of a convolution kernel whose inputs are an 8-bit image and 16-bit 5x5 filter kernel, and whose output is an 8-bit image. On a 32-way SIMD machine, computing 32 5x5 filtered output pixels requires 25 vector MAC operations. If we assume that each vector MAC takes one cycle to complete, and there are no other dependencies, then the SOL performance for this kernel would be 25/32 cyc/pix = 0.78 cyc/pixel. On the VPU, this kernel is executed with 8-bit and 16-bit vector load ops, 16-bit vector MAC ops, and a 16-bit to 8-bit round/saturating store. We are doing the MAC with the vmaddh instruction, which performs 16-bit x 16-bit multiplies into a 16-bit register (where the 8-bit input is promoted to 16-bit during the load for use by the multiply instruction).

The VPU is a 7-way Very Long Instruction Word (VLIW) and Wide-SIMD vector processor. It can run up to 2 scalar operations, 2 vector operations, and 3 memory operations in one execution packet in a single cycle. In this section, we focused on getting a mental count of the operation types in the loop to give us a baseline from which to interpret the results of the loop optimization pragmas that we are reviewing in the later part of this tutorial. In the subsequent sections, we see how the instructions in our convolution loop body can fit into the available instruction slots of an execution packet to reach maximum performance.

  1. Below is the vector “for” loop declaration, along with the appending of the chess_unroll_loop() and chess_prepare_for_pipelining() pragmas. These pragmas give the compiler additional information on the behavior of the loop, which may allow it to produce a more optimal schedule.

        for (int i = 0; i < agens.niter; i++)
        chess_unroll_loop(2) /* Adjust this factor to see differences in loop performance */
    chess_prepare_for_pipelining
    {

  2. The following two load operations constitute 2 out of the 3 total memory ops for this loop.

            dvdataInH = vuchar_dvshortx_load(in1);
        vcoef     = vshort_load_hs(in2);

  3. The following two vector operations constitute the totality of vector ops for this loop. The vector arithmetic operations are comprised of multiply accumulates between the input pixel and filter coefficients.

            dvacc.lo = vmaddh(dvdataInH.lo, vcoef, dvacc.lo, 0, pred_madd);
        dvacc.hi = vmaddh(dvdataInH.hi, vcoef, dvacc.hi, 0, pred_madd);

  4. The following store operation constitutes 1 out of the 3 total memory ops for this loop.

            vstore_hb(dvacc, out, pred_store); // Note: vstore_hb supported only on Gen2 or above devices

  5. The following two scalar operations constitute the totality of scalar ops for this loop.

            pred_madd    = mod_inc(pred_madd, agens.niter_acc_reset - 1);
        s_ctrl_store = mod_inc_pred_z(s_ctrl_store, agens.niter_acc_reset - 1, pred_store);
    }

    The scalar operations compute the predicate variables pred_madd and s_ctrl_store. pred_madd is used to control when the accumulation register is cleared. s_ctrl_store is used to ensure that the accumulator output is stored only at the end of the filter kernel inner loop. Note that the VPU has a specific scalar instruction to do these predicate calculations (modinc_notp, modinc) so that these types of common periodic predicate calculations can be done in one op.

To summarize the above, the convolution loop body gives us 2 scalar operations, 3 memory operations, and 2 vector operations. In theory these could all fit into the same VLIW execution packet and execute in a single clock cycle, as our earlier SOL calculations suggest. However, with the loop in its current form (assuming no usage of the pipelining and unroll pragmas) this is not the case as there is an instruction-to-instruction dependency coming from the LD -> MAC > STORE sequence of operations. Our SOL calculation is a naive approach which assumes that we can fully hide register dependencies by loop unrolling and pipelining. Doing SOL calculations in this manner is often a good place to start as it places an upper bound on the best possible performance. In this particular case, we expect be able to fit the scalar, vector, and mem ops into the same execution packets so we are not bounded by any particular operation type. In other cases, an application could be bounded by the availability of slots for either scalar, vector, or memory op types.

Examining Vector Code disassembly#

We now examine the dissassembled convolution vector code loop body from VPU Programming Basics Tutorial to evaluate how its schedule compares against the SOL estimate. The dissassembled loop body is as follows:

Both supported compilers emit the VLIW packet structure shown here. Exact register allocation and packet ordering depend on the compiler version and optimization decisions.

VLDH_S *A2++,V2       || DVLDBHU_P *A1++,V0:V1
MODINC R10,P2         || [P2] VMAddH_CA V0,V2,AC0|| [P2] VMAddH_CA V1,V2,AC1
MODINC_NOTP R10,R6,P3 || [P3] DVSTHB_P AC0:AC1,*A0++

You can access this disassembly by going to the following location in your build directory:

YOUR_BUILD_OUTPUT_DIR/samples/public/tutorial/vpu_programming_basics/vpu/vpu_programming_basics_dev_gen2.elf.lst

To locate this kernel loop in the disassembly file, search for the Hardware Loop “RPT” instruction whose loop body has the same vector instructions you used in your kernel’s source code. The body of the repeat loop (RPT) executes instructions 2 cycles after the RPT instruction is issued. The immediate field of RPT encodes the PC difference between the second delay slot (just before entering the loop) and the last packet of the loop.

This convolution loop’s schedule from VPU Programming Basics Tutorial takes 3 cycles to produce 32 intermediate output pixels. It takes 75 cycles to produce the full filter output for 32 pixels. This means that our vector code performance is three times slower than the SOL performance calculated. This is because of the dependency between the load, MAC, and store operations mentioned earlier. Currently we are using at most 3 out of the 7 available instruction slots in each execution packet. In order to use more of the instruction slots in an execution packet, we need to pipeline across iterations. At this stage, the compiler does not have any additional information about this loop that it can use to improve its scheduling. We review the concepts of loop unrolling and software pipelining, which allows us to provide more information to the compiler about this loop’s behavior, which then enables it to to produce a better schedule.

Loop Optimization Techniques#

In this section, we review two techniques that can be applied to the convolution loop to help improve its scheduling by the compiler:

  • Software Pipelining

  • Loop Unrolling

Software Pipelining#

Software pipelining is a compiler optimization technique that runs portions of a loop’s prior, current, and future iterations in the same loop body to help utilize more instruction-level paralellism. For example, this means that in one VPU execution packet, the following things could be happening with the convolution code:

  • The three memory units could be loading the coefficient and data for iteration “n+1”, and storing the filter output from iteration “n-1.”

  • The two vector units could be doing the multiply accumulate operations for iteration “n.”

  • The two scalar units could be computing the predicate values for the store of iteration “n.”

If we could acheive this specific instruction packing in our scheduled loop, we would be at the SOL performance of 1 cycle for each intermediate pixel output. By issuing a single instruction packet that contains instructions for both the previous, next, and current iteration, we are able to maximize usage of the 7 available instruction slots.

Loop Unrolling#

When unrolling a loop, you are replicating the code of the inner loop body multiple times, while reducing the overall iteration count by that same factor. For example, if you unroll a 100-iteration loop by a factor of 2, the loop runs a total of 50 iterations, and the inner loop body is executing iterations “n” and “n+1.” By doing this, we can generate more dense VLIW instruction packets, meaning we dispatch more instructions per cycle. For example, the loop unrolling would allow us to run the MAC operations for iteration n, while performing the load of the the input pixels for iteration n+1. Note that there is a tradeoff with the use of loop unrolling in that the larger the unrolling factor, the larger your resulting code size is in the generated executable. There is an additional limiting factor with loop unrolling in that you may run into register pressure depending on how large your loop body is, the size of your unrolling factor, and the size of your architecture’s register file. The programmer must also be cautious not to unroll by a factor that is incompatible with the number of iterations the loop actually runs. For example, if you have a loop which is supposed to execute 5 times and you unroll by a factor of 2, you may end up executing the loop 6 times.

On VPU, the user can use the following pragma to instruct the compiler to unroll the loop:

  • chess_unroll_loop(n)

This informs the compiler that it can attempt to unroll the loop by a factor of “n.” Selecting the right unrolling factor of a loop depends on how much vacancy there is in a single iteration due to load-to-use latency (minimum amount of cycles required before the result of a load operation is available in registers) and/or vector math operation latency (which is 6 cycles on Gen2 VPU), so we need to experiment with different unrolling factors to find one that works. Refer to Orin_PVA_VPU_Programmers_Guide.pdf in the SDK docs for additional information on pipelining/unrolling strategies.

Compiler Pragmas for Pipelining#

Often, the best performance is acheived when loop unrolling and software pipelining are done together. We enable these types of optimizations via the following two pragmas:

  • chess_unroll_loop(UNROLL_FACTOR) tells the compiler to replicate the loop body “UNROLL_FACTOR” number of times and adjusts the loop iteration count accordingly.

  • chess_prepare_for_pipelining() tells the compiler to fold and schedule the original loop content into multiple iterations.

SW Pipelined Loop with unroll factor of 2#

Below is the vector “for” loop declaration, along with the appending of the chess_unroll_loop() and chess_prepare_for_pipelining() pragmas.

    for (int i = 0; i < agens.niter; i++)
        chess_unroll_loop(2) /* Adjust this factor to see differences in loop performance */
    chess_prepare_for_pipelining
    {

With these pragmas, we are instructing the compiler to unroll the loop once and apply software pipelining. Let’s examine the disassembly output for these settings:

  • Prologue (Computes iteration 0 & 1, and loads inputs for iterations 2 & 3)

ADDI R2,#-2,R2    || MOVSP R0,P3    || DVLDBHU_P *A0++,V0:V1|| VLDH_S *A1++,V4
DVLDBHU_P *A0++,V2:V3|| VLDH_S *A1++,V5
[P3] VMAddH_CA V0,V4,AC0|| [P3] VMAddH_CA V1,V4,AC1|| DVLDBHU_P *A0++,V0:V1|| VLDH_S *A1++,V4
RPT R2,#15
MODINC R4,P3      || NOP            || NOP            || NOP            || NOP            || NOP
MODINC_NOTP R4,R5,P4|| MODINC R4,P3 || [P3] VMAddH_CA V3,V5,AC1|| [P3] VMAddH_CA V2,V5,AC0|| [P2] DVSTHB_P AC0:AC1,*A2++|| DVLDBHU_P *A0++,V2:V3|| VLDH_S *A1++,V5

  • Steady State Loop Body (Computes iterations 2 to n - 5)

MODINC_NOTP R4,R5,P2|| [P3] VMAddH_CA V0,V4,AC0|| [P3] VMAddH_CA V1,V4,AC1|| DVLDBHU_P *A0++,V0:V1|| VLDH_S *A1++,V4|| [P4] DVSTHB_P AC0:AC1,*A2++
[P2] DVSTHB_P AC0:AC1,*A2++
MODINC R4,P3
MODINC R4,P3      || MODINC_NOTP R4,R5,P4|| [P3] VMAddH_CA V2,V5,AC0|| [P3] VMAddH_CA V3,V5,AC1|| DVLDBHU_P *A0++,V2:V3|| VLDH_S *A1++,V5

  • Epilogue (Completes computation for remaining iterations)

MODINC R4,P3      || MODINC_NOTP R4,R5,P2|| [P3] VMAddH_CA V0,V4,AC0|| [P3] VMAddH_CA V1,V4,AC1|| [P4] DVSTHB_P AC0:AC1,*A2++
JR R15
MODINC_NOTP R4,R5,P4|| [P3] VMAddH_CA V2,V5,AC0|| [P3] VMAddH_CA V3,V5,AC1|| [P2] DVSTHB_P AC0:AC1,*A2++
[P4] DVSTHB_P AC0:AC1,*A2++
RPT R2,#14
MOVSP R0,P3
NOP
MODINC_NOTP R4,R5,P4|| VLDH_S *A1++,V4|| DVLDBHU_P *A0++,V0:V1
VLDH_S *A1++,V5   || DVLDBHU_P *A0++,V2:V3
MODINC R4,P3      || [P3] VMAddH_CA V0,V4,AC0|| [P3] VMAddH_CA V1,V4,AC1
MODINC R4,P3      || MODINC_NOTP R4,R5,P2|| [P3] VMAddH_CA V2,V5,AC0|| [P3] VMAddH_CA V3,V5,AC1|| [P2] DVSTHB_P AC0:AC1,*A2++
[P4] DVSTHB_P AC0:AC1,*A2++

In sofware pipelined loops, there is a prologue and epilogue that surrounds the steady state loop body. The prologue queues up the pipeline and the epilogue winds it down. In cases where the loop iteration count is not known at compile time, the prologue/epilogue has many branch statements. These branches can be avoided by specifying chess_loop_range(). Most of the loop iterations are run in the steady state loop body. In this case the steady-state performance is 4 cycles for every 64 intermediate output pixels. This is 2x slower than the SOL, and does not hide the 6-cycle vector math operation latency. In two of the steady state cycles we are executing 6 instructions per execution packet. In the other two cycles of the steady state loop we are executing only one instruction per packet so there is still more room for instruction-level parallelism as we are not utilizing all of the available instruction slots.

SW Pipelined Loop with unroll factor of 8#

We can try to improve the paralellism by increasing the unrolling factor to 8. Below is the disassembly output for this setting:

  • Prologue (Computes iteration 0 & 1, and loads inputs for iterations 2 to 7)

ADDI R2,#-1,R2    || MOVSP R0,P3    || VLDH_S *A1++,V22|| DVLDBHU_P *A0++,V12:V13
VLDH_S *A1++,V20  || DVLDBHU_P *A0++,V8:V9
VLDH_S *A1++,V18  || DVLDBHU_P *A0++,V4:V5
VLDH_S *A1++,V16  || DVLDBHU_P *A0++,V0:V1
VLDH_S *A1++,V17  || DVLDBHU_P *A0++,V2:V3
RPT R2,#56        || NOP            || NOP            || NOP            || VLDH_S *A1++,V19|| DVLDBHU_P *A0++,V6:V7
MODINC R4,P3      || NOP            || [P3] VMAddH_CA V12,V22,AC0|| [P3] VMAddH_CA V13,V22,AC1|| DVLDBHU_P *A0++,V10:V11|| VLDH_S *A1++,V21|| NOP
MODINC_NOTP R4,R5,P4|| MODINC R4,P3 || [P3] VMAddH_CA V9,V20,AC1|| [P3] VMAddH_CA V8,V20,AC0|| [P2] DVSTHB_P AC0:AC1,*A2++|| DVLDBHU_P *A0++,V14:V15|| VLDH_S *A1++,V23

  • Steady State Loop Body (Computes iterations 2 to n - 15)

MODINC R4,P3      || MODINC_NOTP R4,R5,P5|| [P3] VMAddH_CA V4,V18,AC0|| [P3] VMAddH_CA V5,V18,AC1|| [P4] DVSTHB_P AC0:AC1,*A2++|| DVLDBHU_P *A0++,V12:V13|| VLDH_S *A1++,V22
MODINC R4,P3      || MODINC_NOTP R4,R5,P6|| [P3] VMAddH_CA V0,V16,AC0|| [P3] VMAddH_CA V1,V16,AC1|| [P5] DVSTHB_P AC0:AC1,*A2++|| DVLDBHU_P *A0++,V8:V9|| VLDH_S *A1++,V20
MODINC R4,P3      || MODINC_NOTP R4,R5,P7|| [P3] VMAddH_CA V2,V17,AC0|| [P3] VMAddH_CA V3,V17,AC1|| [P6] DVSTHB_P AC0:AC1,*A2++|| DVLDBHU_P *A0++,V4:V5|| VLDH_S *A1++,V18
MODINC R4,P3      || MODINC_NOTP R4,R5,P8|| [P3] VMAddH_CA V6,V19,AC0|| [P3] VMAddH_CA V7,V19,AC1|| [P7] DVSTHB_P AC0:AC1,*A2++|| DVLDBHU_P *A0++,V0:V1|| VLDH_S *A1++,V16
MODINC R4,P3      || MODINC_NOTP R4,R5,P9|| [P3] VMAddH_CA V10,V21,AC0|| [P3] VMAddH_CA V11,V21,AC1|| [P8] DVSTHB_P AC0:AC1,*A2++|| DVLDBHU_P *A0++,V2:V3|| VLDH_S *A1++,V17
MODINC_NOTP R4,R5,P10|| MODINC R4,P3|| [P3] VMAddH_CA V14,V23,AC0|| [P3] VMAddH_CA V15,V23,AC1|| [P9] DVSTHB_P AC0:AC1,*A2++|| DVLDBHU_P *A0++,V6:V7|| VLDH_S *A1++,V19
MODINC R4,P3      || MODINC_NOTP R4,R5,P2|| [P3] VMAddH_CA V12,V22,AC0|| [P3] VMAddH_CA V13,V22,AC1|| [P10] DVSTHB_P AC0:AC1,*A2++|| DVLDBHU_P *A0++,V10:V11|| VLDH_S *A1++,V21
MODINC R4,P3      || MODINC_NOTP R4,R5,P4|| [P3] VMAddH_CA V8,V20,AC0|| [P3] VMAddH_CA V9,V20,AC1|| DVLDBHU_P *A0++,V14:V15|| VLDH_S *A1++,V23|| [P2] DVSTHB_P AC0:AC1,*A2++

  • Epilogue (Completes computation for remaining iterations)

MODINC R4,P3      || MODINC_NOTP R4,R5,P5|| [P3] VMAddH_CA V4,V18,AC0|| [P3] VMAddH_CA V5,V18,AC1|| [P4] DVSTHB_P AC0:AC1,*A2++
MODINC R4,P3      || MODINC_NOTP R4,R5,P6|| [P3] VMAddH_CA V0,V16,AC0|| [P3] VMAddH_CA V1,V16,AC1|| [P5] DVSTHB_P AC0:AC1,*A2++
MODINC R4,P3      || MODINC_NOTP R4,R5,P7|| [P3] VMAddH_CA V2,V17,AC0|| [P3] VMAddH_CA V3,V17,AC1|| [P6] DVSTHB_P AC0:AC1,*A2++
MODINC R4,P3      || MODINC_NOTP R4,R5,P8|| [P3] VMAddH_CA V6,V19,AC0|| [P3] VMAddH_CA V7,V19,AC1|| [P7] DVSTHB_P AC0:AC1,*A2++
MODINC R4,P3      || MODINC_NOTP R4,R5,P9|| [P3] VMAddH_CA V10,V21,AC0|| [P3] VMAddH_CA V11,V21,AC1|| [P8] DVSTHB_P AC0:AC1,*A2++
JR R15
MODINC_NOTP R4,R5,P10|| [P3] VMAddH_CA V14,V23,AC0|| [P3] VMAddH_CA V15,V23,AC1|| [P9] DVSTHB_P AC0:AC1,*A2++
NOP               || [P10] DVSTHB_P AC0:AC1,*A2++
MOVSP R0,P3       || MODINC_NOTP R4,R5,P4|| VLDH_S *A1++,V22|| DVLDBHU_P *A0++,V12:V13
MODINC_NOTP R4,R5,P5|| VLDH_S *A1++,V20|| DVLDBHU_P *A0++,V8:V9
MODINC_NOTP R4,R5,P6|| VLDH_S *A1++,V18|| DVLDBHU_P *A0++,V4:V5
MODINC_NOTP R4,R5,P7|| VLDH_S *A1++,V16|| DVLDBHU_P *A0++,V0:V1
MODINC_NOTP R4,R5,P8|| VLDH_S *A1++,V17|| DVLDBHU_P *A0++,V2:V3
MODINC_NOTP R4,R5,P9|| VLDH_S *A1++,V19|| DVLDBHU_P *A0++,V6:V7
MODINC R4,P3      || [P3] VMAddH_CA V12,V22,AC0|| [P3] VMAddH_CA V13,V22,AC1|| VLDH_S *A1++,V21|| DVLDBHU_P *A0++,V10:V11
MODINC R4,P3      || [P3] VMAddH_CA V8,V20,AC0|| [P3] VMAddH_CA V9,V20,AC1|| [P2] DVSTHB_P AC0:AC1,*A2++|| VLDH_S *A1++,V23|| DVLDBHU_P *A0++,V14:V15
MODINC R4,P3      || [P3] VMAddH_CA V4,V18,AC0|| [P3] VMAddH_CA V5,V18,AC1|| [P4] DVSTHB_P AC0:AC1,*A2++
MODINC R4,P3      || [P3] VMAddH_CA V0,V16,AC0|| [P3] VMAddH_CA V1,V16,AC1|| [P5] DVSTHB_P AC0:AC1,*A2++
MODINC R4,P3      || [P3] VMAddH_CA V2,V17,AC0|| [P3] VMAddH_CA V3,V17,AC1|| [P6] DVSTHB_P AC0:AC1,*A2++
MODINC R4,P3      || [P3] VMAddH_CA V6,V19,AC0|| [P3] VMAddH_CA V7,V19,AC1|| [P7] DVSTHB_P AC0:AC1,*A2++
JR R15            || MODINC R4,P3   || [P3] VMAddH_CA V10,V21,AC0|| [P3] VMAddH_CA V11,V21,AC1|| [P8] DVSTHB_P AC0:AC1,*A2++
MODINC_NOTP R4,R5,P10|| [P3] VMAddH_CA V14,V23,AC0|| [P3] VMAddH_CA V15,V23,AC1|| [P9] DVSTHB_P AC0:AC1,*A2++
[P10] DVSTHB_P AC0:AC1,*A2++

In this case, the steady state loop performance is 8 cycles for 256 intermediate output pixels, or 1 cycle for every 32 intermediate output pixel, which is the SOL performance we computed earlier. Upon examination of the instruction packing in the steady state loop, we can see that each packet is executing 7 instructions in parallel, so we are fully utilizing the 7-way VLIW architecture, and effectively hiding load and vector math operation latencies.

Optimal Use of the Instruction Cache#

Once a kernel’s inner loop is fully packed against the VLIW slots, the next class of optimization opportunity is keeping the hot path resident in the instruction cache (iCache). Address this in a tiered order — confirm the problem first, then prefer the lighter-touch remedies. A custom BCF file is a last resort, not a first move.

Step 1 — Confirm iCache misses are the bottleneck#

A VPU stall in Nsight Systems alone is not enough to diagnose iCache pressure; many other causes (VMEM bank conflicts, DMA sync waits, scalar hazards) produce similar timeline shapes. Confirm with the on-chip perfmon counters described in Profiling Tools. Use the PerfmonCounters device API to read the iCache miss event count for the kernel and compare it to a sibling kernel with known-good locality. Only proceed if the miss rate is materially above noise.

Step 2 — Refactor the source to give the compiler a clean layout#

Most iCache pressure is created by source organization that mixes cold setup code with the hot loop, or by hidden calls into the softfloat library at the end of the instruction segment. Two zero-risk source changes resolve most cases:

  • Separate setup from the hot loop. Move one-time setup into its own function and leave the per-tile main loop alone. The compiler then naturally places the loop body after the setup body, so the cold prologue does not sit inside the working set the hot loop refills on every tile.

  • Avoid softfloat helpers. Scalar floating-point operations such as reciprocal, square root, division, and the transcendentals link against the softfloat library, which is placed at the end of the instruction segment. Jumping to it from a hot loop pulls a faraway cache line on every call. Confirm by searching the .map file for the presence of softfloat.a; if it has been pulled in, replace the offending operations with VPU intrinsics such as frcp() for scalar float reciprocal or fsqrt() for square root.

Re-measure after each refactor before moving to the next step.

Step 3 — Hint the cache from device code with cupvaICachePrefetch#

If source-level refactoring does not eliminate the misses, the cupvaICachePrefetch() device API can prime the cache with the address range of an upcoming hot function. Issue the prefetch a few cycles before the call so the lines arrive before the actual branch. This requires no build-system changes, only a few extra lines in the device code. Re-measure again.

Step 4 — Last resort: control IMEM layout#

Reach for a custom linker input only when steps 1–3 leave iCache misses on the table and the program-memory section of the .map file shows that the compiler-chosen layout does not match the hot-loop control flow. NVVPU uses an lld linker script supplied through the LD field of pva_device; see NVVPU Linker Scripts. ASIP Programmer uses a BCF file supplied through BCF; see Using ASIP Programmer.

Recommended workflow for this step:

  1. Capture a baseline with the default linker input. Record per-tile cycles and the perfmon iCache miss counter from step 1 of the playbook.

  2. Inspect the program-memory section of the .map file to identify the placement of the hot functions and the cold ranges separating them.

  3. Author a custom linker input that includes the SDK default and places the hot functions adjacent in IMEM.

  4. Rebuild and re-measure with the same tile sizes. Confirm that the iCache miss counter falls and per-tile cycles drop. If only the miss counter changes, you have moved the bottleneck without reaching SOL — return to step 1 of the Optimization Playbook.

Always include the default linker input rather than replacing it wholesale; otherwise the output can omit runtime sections required by the host scheduler.