Miscellaneous Features

Server Status

The CloudXR server reports its status via OpenVR to other processes:

auto *pSystem = vr::VR_Init(&eError, vr::VRApplication_Utility);
if (eError == vr::VRInitError_None)
{
   cxrServerState state = (cxrServerState)(pSystem->GetInt32TrackedDeviceProperty(
                 vr::k_unTrackedDeviceIndex_Hmd,
                     (vr::ETrackedDeviceProperty)cxrTrackedDeviceProperty::Prop_CloudXRServerState_Int32));
}

The definition of the cxrServerState enumeration is in CloudXRCommon.h.

Sending User Data to Server Apps

The CloudXR client can send arbitrary data to an app that is running on the server by using the cxrSendInputEvent() client API with cxrGenericUserInputEvent as the argument type.

The data is delivered to the server computer, is exposed as a memory mapped file, and is visible to every process on the server. The following sample shows how you can access the user data, with the appropriate error checking.

HANDLE hUserDataFile = OpenFileMapping(FILE_MAP_READ, FALSE, cxrUserDataFileName);

unsigned char *mappedUserData =
   (unsigned char*)MapViewOfFile(hUserDataFile, FILE_MAP_READ, 0, 0, cxrUserDataMaxSize);

HANDLE hUserDataMutex = CreateMutex(NULL, FALSE, cxrUserDataMutexName);

for (int i = 0; i < 1000; i++)
{
   WaitForSingleObject(hUserDataMutex, INFINITE);
   printf_s("%.*s\n", cxrUserDataMaxSize, mappedUserData);
   ReleaseMutex(hUserDataMutex);

   Sleep(500);
}

UnmapViewOfFile(mappedUserData);
CloseHandle(hUserDataFile);

Foveated Scaling

Foveated scaling was supported starting in CloudXR 2.0. Foveated scaling considers the viewers’ lower acuity in the periphery of the view and downscales those areas with a variable scaling factor where the texels are 1:1 at the center and increasingly subsampled toward the outer edges.

On the client, this feature is enabled using the -f N Command-Line Options, where N is a scale that is applied to the client resolution and represents the targeted size to downscale. A region of that frame will be preserved and not downscaled, and the rest is increasingly scaled as you get closer to the edge of the frame. Therefore, larger scale factors result in a larger streamed frame resolution with more unscaled pixels, and smaller scale factors will have fewer original quality pixels. The recommended default value for N is 50, which means that a 50% scale factor is applied to the provided device resolution. Valid values are 0 to disable or 25-100 to specify the varying levels of downscaling. We’ve generally found that 50, 60, or 70 are sufficient to test with your content and choose acceptable results.

Forward foveation is supported in the server (implemented as a DX11 shader), and inverse foveation is currently supported only in Windows DirectX, CUDA clients, and in Android GLES clients.

There are plans to support devices with eye tracking in a future CloudXR release, and the foveation center will be adjusted live at this time.

Streaming AR Content

To stream CloudXR with AR content, the server application, which is an OpenVR app, provides the main scene in the left eye that contains RGBA data. The alpha channel in the data indicates the regions that should be blended with live camera content. The right eye data is ignored, but because OpenVR requires the submission of both eyes, for best performance we recommend that the application also submit the left eye texture as the right eye texture, by using the same texture handle, or optionally submit a small dummy texture for the right eye.

Sending AR Lighting Information

The Android CloudXR AR client sends estimates for the color, the direction of the primary scene light, and the spherical harmonics for the ambient light. These estimates can be queried by an OpenVR app that is running on the server in the following way:

vr::ETrackedPropertyError error;
m_pHMD->GetArrayTrackedDeviceProperty(
   vr::k_unTrackedDeviceIndex_Hmd,
       (vr::ETrackedDeviceProperty)Prop_ArLightColor_Vector3,
       vr::k_unHmdVector3PropertyTag,
       &m_mainLightColor,
       sizeof(m_mainLightColor),
       &error);
m_pHMD->GetArrayTrackedDeviceProperty(
   vr::k_unTrackedDeviceIndex_Hmd,
       (vr::ETrackedDeviceProperty)Prop_ArLightDirection_Vector3,
       vr::k_unHmdVector3PropertyTag,
       &m_mainLightDir,
       sizeof(m_mainLightDir),
       &error);

Audio Support

CloudXR supports sending and recieving audio data between the server and client. The Windows and Android Sample clients show how this can be achieved using the CloudXR cxrSendAudio() API and the cxrClientCallbacks.RenderAudio() callback.

Note

Sending audio from the client to server is disabled by default. To enable this the -sa client option and the -ra server option must be set. See Command-Line Options for more information.

Pose to Frame Correlation

CloudXR supports tagging head poses with a user-provided 64-bit unsigned integer value by setting the cxrHmdTrackingFlags_HasPoseID flag and putting the value in the poseID member of cxrHmdTrackingState, and the same value can be seen in the poseID member of the cxrFramesLatched for the frames corresponding to the head pose. This is intended to help client apps identify which exact head pose was used for a rendered server frame.

CloudXR File Storage

There are many points where CloudXR needs to output logs or other data, and thus needs an ‘output directory’ to work from. Historically CloudXR has always used hard-coded default directories. With newer releases, we have moved to the application ‘owning’ directory paths, and just letting CloudXR know where to put things. From the client side, it is set in the appOutputPath member of the cxrReceiverDesc structure passed into cxrCreateReceiver(). In general, we recommend that this path be to something like a logs subdirectory, but the naming and location is up to you.

Windows Storage

On Windows, the sample code uses standard methods to find the AppData path, and then makes an vendor and application specific path, with a logs subdirectory underneath. We generally use an NVIDIA base folder with a CloudXR subfolder as the root, and then the app-specific folder based on the client name. It looks something like this:

char userDir[CXR_MAX_PATH + 1] = "";
std::string appBaseDir, appOutputDir;
HRESULT result = SHGetFolderPath(NULL, CSIDL_LOCAL_APPDATA, NULL, SHGFP_TYPE_CURRENT, userDir);
if (!SUCCEEDED(result) && !GetTempPath(sizeof(userDir), userDir))
    strcpy(userDir, "C:\Temp"); // super-fallback case.
appBaseDir = userDir;
appBaseDir += "\NVIDIA\CloudXR\SampleClient\";
appOutputDir = appBaseDir + "logs\";

After that, you are responsible for ensuring that full path is created and accessible.

Before calling cxrCreateReceiver(), as part of the receiver descriptor you supply the output path you have chosen. Something like:

strncpy(desc.appOutputPath, appOutputDir.c_str(), CXR_MAX_PATH - 1);
desc.appOutputPath[CXR_MAX_PATH - 1] = 0;

That is where all of the other log, trace and capture files will be written.

Android Scoped Storage

While the new approach to passing the output directory to the receiver shifts more control to applications across all platforms, a driving force for this change was Android’s new [https://developer.android.com/training/data-storage#scoped-storage] Scoped Storage feature, which restricts (or ‘Sandboxes’) where in the file system applications can read/write. While initially introduced in Android 10 as an option, in Android 11 it was enforce by default (if on Android 11 with API level <29, there is a way to request ‘legacy’ access, see https://developer.android.com/about/versions/11/privacy/storage#scoped-storage).

As of Android 12, Scoped Storage is now 100% enforced for all applications. This means that we are restricted to the application’s data directory (or in media directories), and we can no longer directly access the root of sdcard. This has impact on reading a CloudXRLaunchOptions.txt file for setting runtime options, and for writing various logging and captures to an output logs folder – both must be done within the application’s directory. However, there is a catch: the app data directory isn’t created until the first run of the application for security reasons. So if you need to use the launch options file approach, you must run your application once, and cleanly exit. After that point, then you can copy a launch options file to the app data directory, or instruct CloudXR the path where it can write output files. From then on, CloudXR functions essentially like prior releases, just restricted to its folder.

Handling Android Data

On newer Android OS releases where scoped storage is enforced, we need the application to locate the app data directory, and supply it to CloudXR during cxrCreateReceiver(). The app data directory can be acquired in many different ways, below are two methods, one for Java, and one for pure native. Again the actual folder doesn’t exist until the application’s first launch.

In a pure native app where you implement main, and have access to the raw android_app structure (the ARCore and OVR samples do), you can trivially get the app data path as it is a member of the Activity struct. You would do something like the following:

std::string outPath = mAndroidApp->activity->externalDataPath;
outPath += "/logs/";
strncpy(desc.appOutputPath, outPath.c_str(), CXR_MAX_PATH - 1);
desc.appOutputPath[CXR_MAX_PATH - 1] = 0;

If instead you are in a Java-centric app, or a native app under an SDK that hides main() from you and thus you have no access to android_app, you will need to pass the path from Java to Native using a JNI call. In your Java main activity, in the OnCreate` method you can call the following to get the app path and pass it along:

path = getExternalFilesDir(null).getAbsolutePath();
someJniNativeFunction(path);

The ArCore sample app has an example of this approach in OnCreate, in the call to :c:func:createNativeApplication. It does it on one line, passing the absolute path from Java down through JNI functions into the native C code, which does all the manipulation to get the Java string into a C string, and passes that into the main application object constructor. The application caches that app path, and creates an ‘output path’ by simply appending “/logs/” to the app path.

All of the samples use similar handling to later load the CloudXRLaunchOptions.txt file, referencing the base app path. This gives apps the flexibility to set their root and output paths as desired, within the access restrictions set by each OS. As above, the output path is then passed into cxrCreateReceiver() so CloudXR knows where the native network library can write its logs, and where CloudXR can output QoS data, video stream captures, frame dumps, and any other debug/trace files.

Handling iOS Data

As is frquently the case, iOS is slightly different across the board. Having a custom readable/writeable directory exposed can be difficult to set up and register properly for user access.

Instead of a wholly custom directory/tree, the iOS sample code simply uses the base documents directory as its root, and adds a logs subfolder there. You can find that code in CXRLogger.swift, and it looks like:

appBaseDirURL = FileManager.default.urls(for: .documentDirectory, in: .userDomainMask).first!
appLogDirURL = appBaseDirURL.appendingPathComponent("logs")
do {
    try FileManager.default.createDirectory(at: appLogDirURL, withIntermediateDirectories: true, attributes: nil)
} catch {
    logMsg(level: cxrLL_Error, tag: LOG_TAG, msg: "Failed to create the app logs directory, logging to console only.")
    return
}

Clearly, you could expand the base directory to append any hierarchy you wish to create, this is just a simplified sample. When we are about to create the receiver, we need to pass in the log/output directory. For the sample, we use somewhat tricky Swift code to take the appLogDirURL from above, and marshal pointers and copy string data into our C data structure. It looks like:

let nsstring = cLogger.appLogDirURL.path as NSString
if let cstr = nsstring.utf8String {
    let _ = withUnsafeMutablePointer(to: &desc.appOutputPath) {
        $0.withMemoryRebound(to: Int8.self, capacity: Int(CXR_MAX_PATH-1)) { ptr in
            strncpy( ptr, cstr, Int(CXR_MAX_PATH-1))
        }
    }
}

How to Launch on Android

The various configurable launch options are covered in depth in Command-Line Options, as are most of these methods. All three of the methods below use the same example launch options in order to more clearly understand the difference in approaches. Also, if you look through the samples you’ll note they generally try to read the launch options file first, then apply any options passed into the application via other means – this allows for ‘baked’ settings on a device to be overridden by command-line (or otherwise provided means).

For the examples below, we will use a fake project name of cxrtest (the APK name), and the fully qualified name we will assume to be com.nvidia.cxrtest.

Launching from ADB

From ADB, you end up running am start to launch your application.

adb shell am start -S -D -n cxrtest.app/com.nvidia.cxrtest.LaunchActivity --es args '"-s 192.168.1.1 -f 50"'

You need to specify which Activity you want to start, in this case LaunchActivity, and then can supply command-line options to pass to the application inside the oddly-quoted string at the end.

Launching from Android Studio

From within Android Studio, you can set command-line options from the “Run/Debug Configurations” dialog. There is a text box titled “Launch Flags”, and you can specify arguments that are passed to the underlying am start command that Android Studio does behind the scenes. Extending from the prior example, you’d input in the “Launch Flags” text box:

--es args "-s 192.168.1.1 -f 50"

Note that this does not need the command string to be wrapped in extra single quotes, that is only needed for manually launching via adb shell.

Launching directly on device

When an app is first installed, it is in a ‘stopped’ state, and cannot receive any automatic intents until manually started by the user. Also note that if you’ve never run the app before on a given device, the app data directory does not yet exist. If you want to use a launch options file, you must run the app once (and exit if it does not do so on its own), so that the data directory will be created by Android and available to you.

Once the app data directory exists, you can create on your PC a file named CloudXRLaunchOptions.txt. That file can contain any of the command-line launch options, either as one long line like the above examples, or for better legibility you can split each option onto its own separate line. You do still need to use the dash prefix before each option.

Then, simply copy the launch options file to the app files directory (for our example, this would be /sdcard/Android/data/com.nvidia.cxrtest/files), and from then on, assuming you use a single server, you can launch from the device without further PC connection. You can drag and drop it using your PC’s file explorer (depending on your OS of choice), or you can use an adb command like:

adb push CloudXRLaunchOptions.txt /sdcard/Android/data/com.nvidia.cxrtest/files/

How to Launch on Windows

This will parallel Android to some extent. Again, the launch options can be found in Command-Line Options. We will look at similar three approaches to launching on windows.

Launching from Command Prompt

Similar but simpler than android, you can launch a client from a command prompt directly, with whatever options you want specified. As a raw example, I’ll use the name ‘cxrtest.exe’ as the application, just to match the above android description.

cxrtest -s 192.168.1.1 -f 50

It’s that simple.

Launching from Visual Studio

From within Visual Studio, you can set command-line options from the project Property pages. If you right-click the project, and select Properties at the end, you’ll get the Property dialog. Select Debugging section from left list. On the right, you’ll now see “Command Arguments”. In there, you can enter the raw command-line to pass to the application, like -s 192.168.1.1 -f 50. Again here you may find it useful to keep some options available but ignore by suffixing with ‘ZZZ’ or something.

Launching with Options File

There is always a time where you want to enter things once, and have it just work. In this case, you can see in the SampleClient in the main() function where it attempts to load and parse CloudXRLaunchOptions.txt. (And then, it follows with attempting to parse any command-line args.) Since the file path is in your control, you can within reason point it anywhere to find the file – but we recommend using the LOCAL_APPDATA path from Windows API, and then append “\NVIDIA\CloudXR\<SampleAppName>”.

As noted prior, you can either put a command-line on one line in the file, or put each command on a separate line, whichever is best for you. You may find the file more understandable to also use the long-format command names and not the 1-3 character abbreviations. They must all still be prefixed with a dash, just like on the command-line. The same parsing code is used in both cases.

Note that your app controls the order and combination of options parsing. So most of the samples you will see try to load the options file first, and regardless of ‘success’ will then go and parse the command-line. This approach has generally proven out to give the most functionality with minimal code.

How to Launch on Linux

If you are building an Android app on Linux instead of Windows, please go follow the instructions in How to Launch on Android.

If you are building a linux client, the above Windows steps are pretty similar. We’ll leave out instruction for launching from an IDE, as on linux there are lots of options, and in many cases developers are just using editors with direct shell access. We will just cover command-line and file topics here.

Launching from Linux Shell

Like Windows, you can launch a client from a shell directly, with whatever options you want specified. As a raw example, I’ll use the name ‘cxrtest’ as the application, just to match the prior descriptions.

./cxrtest -s 192.168.1.1 -f 50

You can throw any of the client command-line options in there.

Launching with Linux Options File

Just like Windows you may prefer to set options in a file once and not need to specify on command-line. The same SampleClient builds for Windows and Linux, so this handling is very similar. In the main() function where it attempts to load and parse CloudXRLaunchOptions.txt. (And then, it follows with attempting to parse any command-line args.) Since the file path is in your control, you can within reason point it anywhere to find the file – but we recommend using the $HOME environment variable, and then append something like “/.CloudXR/SampleClient/” for the app data root. You’ll see the Windows and Linux variants of building a base app directory.

Again, the launch options file can either be a single line with multiple options as if it was a command-line, or you can break each command onto a separate line. You may find the file more understandable to also use the long-format command names and not the 1-3 character abbreviations – and in combination with one option per line, it makes complex options files easier to maintain. All options must still be prefixed with a dash, just like on the command-line, as the same parsing code is used in both cases.

Most of the samples try to load the options file first, and regardless of ‘success’ will then go and attempt to parse any command-line arguments. This approach has generally proven out to give the most functionality with minimal code, and allows you to override some feature in your launch options file for a given run by specifying it on the command-line.