NVIDIA CloudXR Client SDK Files¶

The {sdk-root-folder}\Client folder contains the header file and library files that comprise the NVIDIA CloudXR SDK. You must include the header file in your application code and link to the appropriate library file when you compile your application.

File	File Name Including Path
Header files	{sdk-root-folder}\Client\Include\CloudXRClient.h {sdk-root-folder}\Client\Include\CloudXRCommon.h {sdk-root-folder}\Client\Include\CloudXRInputEvents.h
Windows OS library file	{sdk-root-folder}\Client\Lib\Windows\CloudXRClient.lib
Android OS library file	{sdk-root-folder}\Client\Lib\Android\CloudXR.aar
iOS library file	{sdk-root-folder}\Client\Lib\iOS\libCloudXR.a
iOS StreamSDK framework	{sdk-root-folder}\Client\Lib\iOS\StreamSdk.framework

To create a new application on most platforms, copy or reference the libraries/headers in the application project or makefile.

For Android, copy the CloudXR.AAR file into your project directories, typically the {project-root}\app\libs directory, and the build.gradle script can unpack this file. This unpacking process generates the required headers and libraries, so you do not need to copy or reference the base headers. For more information, look at the existing CloudXR Android samples.

Sample Client Application Source Code¶

The {sdk-root-folder}\Sample folder contains source code and the associated third-party code to build sample applications for supported client hardware on Windows, Android, and iOS operating systems. The sample applications include:

Developing a CloudXR Client¶

Overview¶

At a high level, the following phases exist in a CloudXR client application:

Client Setup: Creates the client Receiver object and initiate server connection.
Client Main Loop: Latches and renders frames, releases the frames, and handle state changes, such as connecting and disconnecting.
Client Cleanup: Cleans up threads and all CloudXR resources, which releases the Receiver object.

A simplistic main() function in psuedo-code might look like the following:

int main() {
   MySetupDeviceDesc(&ddesc);
   MySetupCallbacks(&callbacks);
   MySetupReceiverDesc(&rdesc, ddesc, callbacks)

   cxrCreateReceiver();
   cxrConnect();

   while (!*exiting*) {
      MyPlatformEventHandling();
      if (client_state < connected) {
         MyRenderConnectionProgress();
      }
      else if (client_state == connected) {
         cxrLatchFrame()
         MyRenderFrames(framesLatched);
         // NOTE: on Android you must call cxrBlitFrame()
         cxrReleaseFrame()
         // NOTE: every n frames you may call cxrGetConnectionStats()
      }
      else {
         MyHandleDisconnect();
         exiting = true;
      }
   }

   cxrDestroyReceiver()
}

Client Setup¶

To set up the client:

Allocate a device descriptor structure (see cxrDeviceDesc). This allocation contains details about the hardware device properties or other specific runtime settings, including information such as the type of data to stream (see cxrDeliveryType), the device/stream resolution and FPS, audio support, and frequency to poll input.

As an example, the Windows sample client has a member struct allocated as part of its window class, and fills in fields with information like the following:

m_deviceDesc.deliveryType = cxrDeliveryType_Stereo_RGB;

d->GetWindowBounds(&x, &y, &m_deviceDesc.width, &m_deviceDesc.height);
m_deviceDesc.width /= 2;
m_deviceDesc.maxResFactor = m_clientOptions.mMaxResFactor;

m_deviceDesc.fps = m_hmd->GetFloatTrackedDeviceProperty(k_unTrackedDeviceIndex_Hmd, Prop_DisplayFrequency_Float);
m_deviceDesc.ipd = m_hmd->GetFloatTrackedDeviceProperty(k_unTrackedDeviceIndex_Hmd, Prop_UserIpdMeters_Float);

m_deviceDesc.predOffset = 0;
m_deviceDesc.proj[0][0] = m_deviceDesc.proj[0][2] = m_deviceDesc.proj[1][0] = m_deviceDesc.proj[1][2] = -1;
m_deviceDesc.proj[0][1] = m_deviceDesc.proj[0][3] = m_deviceDesc.proj[1][1] = m_deviceDesc.proj[1][3] = 1;

m_deviceDesc.audio = m_clientOptions.mReceiveAudio;
m_deviceDesc.sendAudio = m_clientOptions.mSendAudio;

m_deviceDesc.embedInfoInVideo = false;
m_deviceDesc.posePollFreq = 0;
m_deviceDesc.disablePosePrediction = false;
m_deviceDesc.angularVelocityInDeviceSpace = false;

m_deviceDesc.foveatedScaleFactor = m_clientOptions.mFoveation;
m_deviceDesc.foveationModeCaps = 0;

m_deviceDesc.ctrlType = cxrControllerType_OculusTouch;

GetChaperone(&m_deviceDesc.chaperone);

Set up a client callback struct (see cxrClientCallbacks), which holds pointers to callback functions that your application supports.

The GetTrackingState callback must be implemented if the application wants the server to sync with latest view and input changes. If the client application wants to support playing back audio from the server, implement the RenderAudio callback and pass along audio buffers to some audio playback system – see the samples for a few approaches across platforms. Finally, implement the UpdateClientState callback to be notified of unexpected disconnects during streaming and to track state changes while connecting to a server.
Prepare a receiver descriptor struct (see cxrReceiverDesc).

This step copies the device descriptor and the client callbacks, sets the client context for all callbacks, which might be a singleton object pointer, or other global struct, selects the XR mode (XR, VR, AR) or Generic mode (individual, possibly unrelated, streams), and sets various debug/logging options.

Filling out the receiver descriptor might look like the following for a Windows client:
```
m_receiverDesc.requestedVersion = CLOUDXR_VERSION_DWORD;

m_receiverDesc.deviceDesc = m_deviceDesc;
m_receiverDesc.clientCallbacks = m_clientCallbacks;
m_receiverDesc.clientContext = this;
m_receiverDesc.shareContext = nullptr;

m_receiverDesc.receiverMode = cxrStreamingMode_XR;
m_receiverDesc.numStreams = CXR_NUM_VIDEO_STREAMS_XR;

m_receiverDesc.debugFlags = m_clientOptions.mDebugFlags;
m_receiverDesc.logMaxSizeKB = m_clientOptions.mLogMaxSizeKB;
```
With all of the structures and fields prepared, the client can now call cxrCreateReceiver(), passing in the receiver descriptor, and a pointer to a cxrReceiverHandle to hold the returned Receiver handle needed for all further interactions with the CloudXR SDK. If it fails, report the error to the user (and log it), and exit cleanly.
If the creation of the Receiver succeeded, initiate a connection to the server by using cxrConnect().

The parameters are the Receiver object, the IP address of the server to connect to, and connection flags. The IP is presumed to be a dotted numeric IPv4 address, which might have been manually entered, translated using something like DNS or custom matchmaking, or gathered from something like mDNS/Bonjour. The connection flags field currently has one option, ConnectAsync.

cxrConnectionFlags_ConnectAsync> instructs the library to use a background thread to initiate connection to the server, instead of on the current thread. If you are using the asynchronous mode, ensure that you implement the handling of connection state changes in an UpdateClientState callback.
If the connection fails, notify the user about the error and exit or return to your connection UI to allow the user to try again.
If the connection succeeds, set a state in the application to indicate that streaming is ready, and in the main loop, handle the streaming status change and begin rendering the frames.

Client Main Loop¶

Client States¶

The main loop of the application might need to deal with different states and determine what to do and render in each state.

Note

Many of the states map directly to values in cxrClientState.

Before receiver creation

If the main loop includes more than just CloudXR streaming, and does not instantiate the CloudXR client until some state is achieved, the application might be rendering the UI to interact with the user, or just show a loading indicator.
Before the connection

The application might show a loading indicator, or in the case of connecting to the server asynchronously, it might show a Connecting to server indicator.
After a successful connection

When cxrConnect() is called with the asynchronous flag, or called in synchronous mode but in a background thread, the main loop needs to recognize the transition into successful streaming with a flag or a state variable. An application can then begin a fade-in transition or display a Connection Established message.
Render streamed frames

After the application has established a connection to the server, it is ready to start receiving video frames from the network. Each time through the main loop while connected, the code needs to determine whether there are frames available. After this, the latest frames are retrieved, rendered out, and released back to the system. This process is covered in the sections below.
After the disconnect

After a disconnect state is detected, the application might need to set flags or make calls to indicate to other systems that streaming has finished, and that anything related to the CloudXR session needs to be cleaned up and shut down. If an unexpected disconnect occurred, the application might display an error message and exit or return to the initial connection interface.

Rendering Streaming Video¶

Frame Acquire¶

To determine whether there is an available video frame, call cxrLatchFrame() to attempt to acquire the next frame(s) in order. The first parameter of interest is the cxrFramesLatched structure, which needs to have a scope so that it will exist until the rendering is complete. After being returned, it is populated with information about the frame(s) that have been acquired.

The next paramter is a bitmask for the frames/streams from which to acquire frames. Most applications can just pass cxrFrameMask_All to tell the system to acquire frames from all streams in lockstep. This is typically the case for XR/AR/VR but will work in most situations for Generic mode connections.

However, some Generic mode applications might want one stream at a time, so they will loop over the total video streams by index, and can latch one at a time passing as the mask 1<<index each time. They can also get a specific ‘subset’ of streams if the particular indices are well known, just by OR’ing the stream bitmasks together. For example, to grab streams 0 and 3 you need to pass 1<<0 | 1<<3.

The last parameter to LatchFrame is a timeout value in milliseconds. If frames are not ready, the parameter will sleep briefly, check again, and repeat this process until the timeout has been exceeded. In general, the timeout will be a factor of the length of a frame, and a reasonable starting value is half display refresh (or 2000/displayHz). This value allows the call to return if too much time has passed without the frame(s) being available. This way, if there is a delay in frame delivery, the application can give cycles to other systems in the main loop. Short timeout values also provide the opportunity to render some cached visual to the screen and/or an indication of a streaming delay. If the application prefers to manually manage sleeps, a timeout of zero will result in a check and quick return without any sleep.

Frame Render¶

If the Latch fails, the application can either skip rendering or render something cached, and then continue on through the main loop. This gives other systems a chance to run/update in the event of a frame delay, and ensures any per-frame state-checking logic (such as handling input, or client state changes like disconnects) occurs in a reasonable period of time.

If the Latch succeeds, then the returned cxrFramesLatched structure holds the frame data needed to render out the frame(s). For Android, there is an API call cxrBlitFrame() that should be called after setting up render target and viewport, and it will use the shared OpenGL|ES context to properly blit out latched frames (including handling things like alpha-blending for AR streams, or de-foveation for VR streams). For other platforms, portions of post-processing like de-foveation is handled during the decoding step, and it is then up to the application to know what data format the decoded frame data is in and how to render (blit/submit) as appropriate for the given graphics API.

Frame Release¶

When rendering is completed, cxrReleaseFrame() must be called to tell CloudXR that you have consumed the latched frame(s), and that CloudXR can internally release and recycle.

Updating Headset Properties¶

To update the headset projection parameters, refresh rate, or IPD alongside a pose tracking update, the HasProjection, HasRefresh, or HasIPD must be specified and the new projection parameters, refresh rate, or IPD value set in cxrHmdTrackingState’s proj, displayRefresh, or ipd fields.

Connection Stats¶

Periodically cxrGetConnectionStats() may be called to monitor the health of the connection. We recommend waiting for fps * 3 frames to be latched (~3 sec) between calls. Examples of how to do this and how to interpet the stats have been implemented in the sample clients.

Client Cleanup¶

Before exiting your application, free resources that are connected to CloudXR. At a minimum, you must call cxrDestroyReceiver(), which will flush internal buffers and shared handles.