Dynamic Remap

Overview

Dynamic Remap applies a generic geometrical transformation to an image. It extends the existing Remap by allowing:

1. Transformation map as an image
2. Map as input argument to the algorithm
3. Different interpolation types for the map
4. The use of relative map for map values
5. The use of normalized map for map values
6. Asynchronous change of the map in algorithm sequence
7. Different sizes for source (input), destination (output) and map images

The example below shows an use of dynamic remap to warp a fisheye image with a simple map.

Source	Map	Destination
Copyright © 2012 by Michel Thoby, with permission from author.	source interpolation: linear map interpolation: bi-cubic flags: VPI_RELATIVE_MAP | VPI_NORMALIZED_MAP map contents: (+0.15, +0.15) (-0.15, +0.15) (+0.15, -0.15) (-0.15, -0.15)

Implementation

The algorithm writes in the destination image a pixel read in the source image in a position determined by the map image. For each destination pixel, the algorithm reads a map value at its corresponding position using the map interpolation type. The map value is either an absolute position \( (x, y) \), denormalized (when using zero in algorithm flags) or normalized (when setting VPI_NORMALIZED_MAP in flags), or a relative offset (when also setting VPI_RELATIVE_MAP) \( (dx, dy) \) from the normalized destination coordinate to a normalized position in the source (see illustration below). The pixel at that source position is fetched from source and stored at the destination position. The sampling of map values and source pixels is controlled by map and source interpolation types and also by algorithm flags. By using VPI_SAMPLING_ALIGNED_TO_CORNER in flags, the sampling occurs at the top-left corner of the pixels instead of the pixel center.

The illustration above explains how the dynamic remap algorithm works. All three images (source, map and destination) can have different sizes. The destination image controls the sampling in the map, i.e. each pixel in destination samples one pixel in the map by normalizing the destination coordinate and using the map interpolation type.

The map stores control points as two floating-point coordinates (see VPI_IMAGE_FORMAT_2F32 and VPIKeypointF32) as either \( (x, y) \) or \( (dx, dy) \). Absolute coordinates \( (x, y) \) are direct pixel positions in the source image. Relative coordinates \( (dx, dy) \) are delta differences that will be added to the map current position to yield the target source position to fetch the pixel from. When using relative map, it is required to use normalized map as well.

The images above shows an example of relative and normalized mapping, using bi-cubic (see VPI_INTERP_CATMULL_ROM) map interpolation type. Using these parameters, a simple \( 2x2 \) map is capable of remapping the source image into the destination image. Furthermore, both the map and source images can be modified in subsequent calls of the algorithm, for example during video frames or panorama stiching.

C API functions

For list of limitations, constraints and backends that implements the algorithm, consult reference documentation of the following functions:

Function	Description
vpiCreateDynamicRemap	Create a payload for Dynamic Remap algorithm.
vpiSubmitDynamicRemap	Submits a Dynamic Remap operation to the stream.

Usage

Language: C/C++ Python

1. Import VPI module

   import vpi

2. Define output size as 80% of the input size.

   dst_size = (int(input.size[0] * .8), int(input.size[1] * .8))

3. Define a map as a 2x2 image with 2F32 image format.

   map = vpi.Image((2, 2), vpi.Format._2F32)
   with map.rwlock_cpu() as data:
       data[0, 0] = [.15, .15]
       data[0, 1] = [-.15, .15]
       data[1, 0] = [.15, -.15]
       data[1, 1] = [-.15, -.15]

4. Executes the algorithm, it produces the output image from the input and map.

   output = vpi.dynamic_remap(input, map, size=dst_size, map_interp=vpi.Interp.CATMULL_ROM,
                              relative_map=True, backend=vpi.Backend.CUDA)

1. Initialization phase
   1. Include the header that defines the Dynamic Remap functions.

      #include <vpi/algo/DynamicRemap.h>

   2. Define the source (or input) image object.

          VPIImage src = /*...*/;

   3. Create the destination (or output) image. In this particular case, destination has 80% the size of the source image.

          int32_t srcSize[2];
          vpiImageGetSize(src, &srcSize[0], &srcSize[1]);
       
          VPIImageFormat srcFormat;
          vpiImageGetFormat(src, &srcFormat);
       
          int32_t dstSize[2] = {(int32_t)(srcSize[0] * .8), (int32_t)(srcSize[1] * .8)};
       
          VPIImage dst;
          vpiImageCreate(dstSize[0], dstSize[1], srcFormat, 0, &dst);

   4. Create the stream where the algorithm will be submitted for execution.

          VPIStream stream;
          vpiStreamCreate(0, &stream);

   5. Generate a custom mapping to warp a fisheye image into a more regular image. The map image stores the control points, corresponding to normalized relative offsets.

          int32_t mapSize[2] = {2, 2};
       
          VPIImage map;
          vpiImageCreate(mapSize[0], mapSize[1], VPI_IMAGE_FORMAT_2F32, 0, &map);
       
          VPIImageData mapData;
          vpiImageLockData(map, VPI_LOCK_WRITE, VPI_IMAGE_BUFFER_HOST_PITCH_LINEAR, &mapData);
       
          VPIImagePlanePitchLinear mapPlane = mapData.buffer.pitch.planes[0];
       
          VPIKeypointF32 *row0 = (VPIKeypointF32 *)((uint8_t *)mapPlane.data + mapPlane.pitchBytes * 0);
          VPIKeypointF32 *row1 = (VPIKeypointF32 *)((uint8_t *)mapPlane.data + mapPlane.pitchBytes * 1);
       
          row0[0] = (VPIKeypointF32){.x = +.15f, .y = +.15};
          row0[1] = (VPIKeypointF32){.x = -.15f, .y = +.15};
          row1[0] = (VPIKeypointF32){.x = +.15f, .y = -.15};
          row1[1] = (VPIKeypointF32){.x = -.15f, .y = -.15};
       
          vpiImageUnlock(map);

   6. Create the algorithm payload. The payload is created in the CUDA backend.

          VPIPayload payload;
          vpiCreateDynamicRemap(VPI_BACKEND_CUDA, &payload);

2. Processing phase
   1. Submit the algorithm to the stream, along with all parameters. The algorithm will be executed by the backend where the payload was created, i.e. CUDA.

          VPI_CHECK_STATUS(vpiSubmitDynamicRemap(stream, VPI_BACKEND_CUDA, payload, src, dst, map, VPI_INTERP_LINEAR,
                                                 VPI_INTERP_CATMULL_ROM, VPI_BORDER_REFLECT,
                                                 VPI_RELATIVE_MAP | VPI_NORMALIZED_MAP));

   2. Optionally, wait until the processing is done.

          vpiStreamSync(stream);

3. Cleanup phase
   1. Free resources held by the stream, the payload, the source, destination and map images.

          vpiStreamDestroy(stream);
          vpiPayloadDestroy(payload);
          vpiImageDestroy(src);
          vpiImageDestroy(dst);
          vpiImageDestroy(map);

Performance

For information on how to use the performance tables below, see Algorithm Performance Tables.
Before comparing measurements, consult Comparing Algorithm Elapsed Times.
For further information on how performance was benchmarked, see Performance Benchmark.

clear filters Device: Jetson AGX Orin  -  Streams: 1 2 4 8

For more information, see Dynamic Remap in the "C API Reference" section of VPI - Vision Programming Interface.