NVIDIA TXAA 3.0

#### Next topic

TXAA 3.0 Utility Library

# NVIDIA TXAA 3.0¶

## Overview¶

NVIDIA TXAA 3.0 creates a smoother, clearer image than any other anti-aliasing solution by combining high-quality MSAA multisample anti-aliasing, post process, and NVIDIA designed temporal filters. This release provides a temporal super-sampling solution that is known to work with the deferred engines without incurring the high cost of shading at every sample, large G-buffers and the associated bandwidth. The central idea in temporal super sampling is to shade the geometry at different sample locations within a pixel every frame and then average the color values over the last few frames. Shading at different sample locations is achieved by jittering the projection matrix (thus effectively changing sample locations for all pixels). Exponential average instead of a simple average requires only the last frame instead of multiple frames. The last frame’s color includes the contributions from its previous frames and so on. However, when the objects are in relative motion with respect to camera, pixel’s history is found at some other location - the pixel’s position in its previous frame. In such a case, the motion vector at each pixel is used to find where the current pixel was in the previous frame. Application needs to generate the motion vector at each sample location. If the color of the corresponding pixel in the previous frame is very different from the current color, a closest approximation is made using the color space bounding box of its neighborhood and clipping the previous color against that bounding box. This release introduces a newer way of calculating the bounding box using the statistical metrics. This newer way tends to produce tighter bounding boxes resulting in less ghosting but can potentially cause flicker when the camera is not moving. We provide additional knobs to control the flicker.

For more details on temporal super-sampling and statistics based color space bounding box calculations, please follow the references linked in the Features section.

## Package¶

doc/—this documentation page.

extensions/—extra utilities and libraries for use with TXAA.

external/—external dependencies needed for TXAA and samples.

lib/—TXAA import libraries and dlls.

samples/—TXAA sample program.

## Getting Started¶

### Features¶

• Core TXAA modes
• Softer film style resolve + reduction of temporal aliasing. Starting from this release, one can optionally choose to use a wider Blackman-Harris3.3 filter instead of the box filter.
• Temporal super-sampling by jittering the projection matrix.
• User can now choose which motion vector to use within the neighborhood of a pixel to fetch its history, e.g. use the longest motion vector, use the one that is closest to the camera or use the largest and the smallest to find which colors in the history match the best and use that. If the app is using reverse infinite projection or inverse Z-tests, user needs to convey that to the library during the color resolve phase.
• Anti-flicker filter that reduces the flicker in temporal super-sampling mode when there is no motion.
• Tighter color-space bounds based on the statistical metrics that uses the algorithm developed by NVIDIA research.
• Starting with this release, users can choose to do the entire TXAA in RGB space instead of YCoCg. Default is YCoCg.

Temporal Re-projection Modes

• Motion vectors explicitly passed
• Application must generate motion vectors, so potentially more work to integrate
• Ghosting can be eliminated by accounting for object motion and camera motion
• Camera motion extraction via depth and camera matrices (This feature is exposed through TxaaUtil lib, not in the TXAA 3.0 core library.)
• Easy to integrate
• May experience lower AA quality for animating/moving objects due to the lack of motion vectors for them.

MSAA Modes

• 0xMSAA (This is a the default mode)
• 1xMSAA
• 2xMSAA
• 4xMSAA
• 8xMSAA

### Debug Channels¶

The TXAA resolve takes a debug render target that can aid debugging.

View MSAA

• Standard MSAA box filter resolve. Standard MSAA box filter resolve. Optionally Blackman-Harris3.3 filter can be used.
• Optional Anti-flicker filter that reduces temporal flickering (controlled by check-box). If that doesn’t reduce the flicker to an acceptable quality, one can manually loosen the bounding boxes using the bbScale parameter.
• Mode: NV_TXAA_DEBUGCHANNEL_MSAA

View Motion Vectors

• Coloring,
• Light Blue-Purple -> no movement
• Pink -> objects moving left
• Blue -> objects moving up
• Mode: NV_TXAA_DEBUGCHANNEL_MOTIONVECTORS

View Difference Between MSAA and Temporal Re-projection

• Shows difference of current frame and re-projected prior frame
• Current MSAA frame is box filtered
• Used to check if input view projection matrixes are correct
• Should see convergence to a black screen on a static image.
• Should see an outline on edges in motion, and the outline should not grow in size under motion.
• Should see bright areas representing occlusion regions (where the content is only visible in one frame).
• Mode: NV_TXAA_DEBUGCHANNEL_TEMPORALDIFFS

View Temporal Re-projection

• Shows just the temporally reprojected prior frame
• Mode: NV_TXAA_DEBUGCHANNEL_TEMPORALREPROJ

View Control

• Shows the texture used to control the filter softness and the amount of temporal feedback
• Channel: NV_TXAA_DEBUGCHANNEL_CONTROL

### Pipeline¶

TXAA Replaces the MSAA resolve step with this custom resolve pipeline

AxB sized MSAA RGBA color   --------->  T  --> AxB sized RGBA resolved color
AxB sized motion vector     --------->  X                           |
AxB sized RGBA feedback     --------->  A                           |
AXB sized depth                               --------->  A                           |
|                                                                |
+----------------------------------------------------------------+

TXAA needs two AxB sized resolve surfaces (because of the feedback).

NOTE, if the AxB sized RGBA resolved color surface is going to be modified after TXAA, then it needs to be copied, and the unmodified copy needs to be feed back into the next TXAA frame.

NOTE, resolved depth is best resolved for TXAA with the minimum depth of all samples in the pixel. If required one can write min depth for TXAA and max depth for particles in the same resolve pass (if the game needs max depth for something else).

Starting with this release, users can apply jitter offsets of less than one pixel to the projection matrix. This has the same effect as that of moving the visibility and shading samples by the same offsets in the opposite directions. The offsets can be from a low discrepancy sequence e.g. Sobol, and repeat after some number of frames e.g. 16. During the spatial resolve phase, we need to filter the results and the filter weights must be adjusted for these offsets. As a result, these offsets need to be conveyed along with the other resolve parameters.

### SLI Scaling¶

TXAA is a temporal approach to anti-aliasing; as such, it introduces an inter frame dependency on the RGBA feedback surface, which could hinder SLI scaling in AFR mode. An approach for removing the inter frame dependency is to use the NvAPI to manage a separate feedback buffer for each GPU. The TXAA11 sample application that is included with the TXAA library uses the technique to remove the inter frame dependency, improving SLI scaling.

### Use Linear¶

TXAA requires linear RGB color input.

• Use DXGI_FORMAT_*_SRGB for 8-bit/channel colors (LDR); or
• Make sure sRGB->linear (TEX) and linear->sRGB (ROP) conversion is on.
• Use DXGI_FORMAT_R16G16B16A16_FLOAT for HDR.

Tone-mapping the linear HDR data to low-dynamic-range is done in TXAA and TXAA will preserve the linear color space when finishes its pass. For example, after post-processing at the time of color-grading. The hack to tone-map prior to resolve, to workaround the artifacts of the traditional MSAA box filter resolve, is not needed with TXAA. Tone-map prior to resolve will result on wrong colors on thin features.

There is support for compressing some or all of the dynamic range before the resolve using the NvTxaaCompressionRange struct. With TXAA dynamic range compression enabled, the resolved frame will be decompressed back to linear HDR automatically; the application should always supply linear HDR data, and should expect linear HDR results back from TXAA.

### The Motion Vector Field¶

The motion vector field input is FP16, where:

• RG provides a vector {x,y} offset to the location of the pixel in the prior frame.
• The offset is scaled such that {1,0} is a X shift the size of the frame. Or {0,1} is a Y shift the size of the frame.

Suggestions for integration:

• Best case have proper motion vectors.

• Worst case, fetch per-pixel Z and re-transform to prior view in order to generate the motion vector surface.
• In other words at a minimum support camera motion.
• May need to move the motion vector generation to before the MSAA resolve.

• Can use multiple render targets to generate TXAA motion vector field, at the same time as writing a separate vector field for motion blur.
• TXAA and motion blur tend to have different precision requirements.

### GPU API State¶

This library does not save and restore GPU state. All modified GPU state is noted in the API documentation and in the function prototype comments.

## Integration Example¶

Suggested way of TXAA 3.0 integration. Per-pixel motion vector is needed for this method. Value in each pixel is a 2d direction pointing to the previous projected coordinate by the corresponding vertex.

Initialize Context

// DX11
#define __GFSDK_DX11__
#include <GFSDK_TXAA.h>
NvTxaaContextDX11 txaaCtx;
ID3D11Device *dev;
if (NV_TXAA_STATUS_OK != GFSDK_TXAA_DX11_InitializeContext(&txaaCtx, dev)) // No TXAA.

// GL
#define __GFSDK_GL__
#include <GFSDK_TXAA.h>
TxaaCtx txaaCtx;
if (NV_TXAA_STATUS_OK != GFSDK_TXAA_GL_InitializeContext(&txaaCtx)) // No TXAA.

// GL
// This feature is not yet supported in Open GL version of TXAA  3.0.

Replace the MSAA Resolve Step with the TXAA Resolve

Custom Motion Vector

// DX11
NvTxaaContextDX11 *ctx;             // TXAA context.
ID3D11DeviceContext *dxCtx;         // DX11 device context.
ID3D11RenderTargetView *dstRtv;     // Destination texture.
ID3D11ShaderResourceView *mvSrv;    // Source motion vector texture SRV.
ID3D11ShaderResourceView *dstSrv;   // SRV to feedback resolve results from prior frame.
ID3D11ShaderResourceView *ctrlSrv;  // Per pixel control, can be null.
NvTxaaFeedbackParameters fb;        // Feedback parameters
NvTxaaResolveParametersDX11 resolveParameters;
memset(&resolveParameters, 0, sizeof(resolveParameters));
resolveParameters.txaaContext = &m_txaaContext;
resolveParameters.deviceContext = deviceContext;
resolveParameters.resolveTarget = m_resolveTarget.pRTV;
resolveParameters.msaaSource = srv;
resolveParameters.msaaDepth = (m_presetMode == PRESETMODE_0xTXAA) ? m_TXAASceneDepthResolved.pSRV : m_TXAASceneDepth.pSRV;
resolveParameters.feedbackSource = feedback.m_txaaFeedback.pSRV;
resolveParameters.alphaResolveMode = NV_TXAA_ALPHARESOLVEMODE_RESOLVESRCALPHA;
resolveParameters.feedback = &m_feedback;
m_txaaPerFrameConstants.xJitter = pJitter[0];
m_txaaPerFrameConstants.yJitter = pJitter[1];
m_txaaPerFrameConstants.mvScale = m_MVScale;
m_txaaPerFrameConstants.motionVecSelection = NV_TXAA_USEMV_NEAREST; // This can be configured differently.
m_txaaPerFrameConstants.useRGB = 0;
m_txaaPerFrameConstants.frameBlendFactor = GetBlendFactor(); // Manual override of the blend factor.
m_txaaPerFrameConstants.dbg1 = GetDbg1(); // Debugging. Do not modify the default values.
m_txaaPerFrameConstants.bbScale = GetBBScale(); // Scales the bounding box tightness. Default value 1.f
m_txaaPerFrameConstants.enableClipping = GetEnableClipping(); // Disable clipping for debugging.

// GL
NvTxaaContextGL *ctx;             // TXAA context.
GLuint *dst;                      // Destination texture.
GLuint msaaSrc;                   // Source MSAA texture.
GLuint mv;                        // Source motion vector texture.
GLuint ctrl;                      // Per pixel control, can be null.
NvTxaaFeedbackParameters fb;      // Feedback parameters
gfsdk_U32 frameCounter;           // Increment every frame, notice the ping pong of dst.
NvTxaaResolveParametersGL resolveParameters;
memset(&resolveParameters, 0, sizeof(resolveParameters));
resolveParameters.txaaContext = ctx;
resolveParameters.resolveTarget = dst[(frameCounter + 1) & 1;
resolveParameters.msaaSource = msaaSrc;
resolveParameters.feedbackSource = dst[frameCounter & 1];
resolveParameters.controlSource = ctrl;
resolveParameters.alphaResolveMode = NV_TXAA_ALPHARESOLVEMODE_RESOLVESRCALPHA;
resolveParameters.feedback = &fb;
NvTxaaMotionDX11 motion;
memset(&motion, 0, sizeof(motion));
motion.motionVectors = mv;
GFSDK_TXAA_GL_ResolveFromMotionVectors(&resolveParameters, &motion);

Release Context

// DX11
#define __GFSDK_DX11__
#include <GFSDK_TXAA.h>
NvTxaaContextDX11 txaaCtx;
GFSDK_TXAA_DX11_ReleaseContext(&txaaCtx);

// GL
#define __GFSDK_GL__
#include <GFSDK_TXAA.h>
NvTxaaContextGL txaaCtx;
GFSDK_TXAA_GL_ReleaseContext(&txaaCtx);

### TXAA 3.0 Utility Library¶

Besides the TXAA core library, there is also an utility library which provides some stand-alone features that may be helpful in TXAA 3.0 integration.

## Integration Suggestions¶

1. Get the debug pass-through modes working, for instance NV_TXAA_DEBUGCHANNEL_MSAA.

2. Get the TXAA resolve working on the frames without the camera and the scene motion.

2a. Verify motion vector field sign is correct using NV_TXAA_DEBUGCHANNEL_MOTIONVECTORS. Output should be:

Light Blue-Purple -> no movement

Pink -> objects moving left

Blue -> objects moving up

If this is wrong, try the transpose of mvpCurrent and mvpPrior.

2b. Verify motion vector field scale is correct using NV_TXAA_DEBUGCHANNEL_TEMPORALDIFFS. Should see:

No Motion -> edges

Small Motion -> edges maintaining brightness and thickness

If seeing edges increase in size with a simple camera rotation, then motion vector input or mvpCurrent and/or mvpPrior is incorrect.

Also when clipping is disabled, one should see smearing at the locations where there is change in the occlusion. If one sees blurring or other image artifacts, motion vectors can be assumed to be incorrectly used.

2c. Once the motion field is verified, the TXAA resolve should work with motion.