Miscellaneous Features

Server Status

The CloudXR server reports its status via OpenVR to other processes:

auto *pSystem = vr::VR_Init(&eError, vr::VRApplication_Utility);
if (eError == vr::VRInitError_None)
{
   cxrServerState state = (cxrServerState)(pSystem->GetInt32TrackedDeviceProperty(
                 vr::k_unTrackedDeviceIndex_Hmd,
                     (vr::ETrackedDeviceProperty)cxrTrackedDeviceProperty::Prop_CloudXRServerState_Int32));
}

The definition of the cxrServerState enumeration is in CloudXRCommon.h.

Sending User Data to Server Apps

The CloudXR client can send arbitrary data to an app that is running on the server by using the cxrSendInputEvent() client API with cxrGenericUserInputEvent as the argument type.

The data is delivered to the server computer, is exposed as a memory mapped file, and is visible to every process on the server. The following sample shows how you can access the user data, with the appropriate error checking.

HANDLE hUserDataFile = OpenFileMapping(FILE_MAP_READ, FALSE, cxrUserDataFileName);

unsigned char *mappedUserData =
   (unsigned char*)MapViewOfFile(hUserDataFile, FILE_MAP_READ, 0, 0, cxrUserDataMaxSize);

HANDLE hUserDataMutex = CreateMutex(NULL, FALSE, cxrUserDataMutexName);

for (int i = 0; i < 1000; i++)
{
   WaitForSingleObject(hUserDataMutex, INFINITE);
   printf_s("%.*s\n", cxrUserDataMaxSize, mappedUserData);
   ReleaseMutex(hUserDataMutex);

   Sleep(500);
}

UnmapViewOfFile(mappedUserData);
CloseHandle(hUserDataFile);

Foveated Scaling

Foveated scaling was supported starting in CloudXR 2.0. Foveated scaling considers the viewers’ lower acuity in the periphery of the view and downscales those areas with a variable scaling factor where the texels are 1:1 at the center and more subsampled at the outer edges.

On the client, this feature is enabled through the -f N Command-Line Options, where N is a scale that is applied to the client resolution and represents the targeted size to downscale. A region of that frame will be preserved and not downscaled, and the rest is increasingly scaled as you get closer to the edge of the frame. Therefore, larger scale factors result in a larger streamed frame resolution with more unscaled pixels, and smaller scale factors will have fewer original quality pixels. The recommended default value for N is 50, which means that a 50% scale factor is applied to the provided device resolution. Valid values are 0 to disable or 25-100 to specify the varying levels of downscaling.

Forward foveation is supported in the server (implemented as a DX11 shader), and inverse foveation is currently supported only in Windows DirectX, CUDA clients, and in Android GLES clients.

There are plans to support devices with eye tracking in a future CloudXR release, and the foveation center will be adjusted live at this time.

Sending AR Lighting Information

The Android CloudXR AR client sends estimates for the color, the direction of the primary scene light, and the spherical harmonics for the ambient light. These estimates can be queried by an OpenVR app that is running on the server in the following way:

vr::ETrackedPropertyError error;
m_pHMD->GetArrayTrackedDeviceProperty(
   vr::k_unTrackedDeviceIndex_Hmd,
       (vr::ETrackedDeviceProperty)Prop_ArLightColor_Vector3,
       vr::k_unHmdVector3PropertyTag,
       &m_mainLightColor,
       sizeof(m_mainLightColor),
       &error);
m_pHMD->GetArrayTrackedDeviceProperty(
   vr::k_unTrackedDeviceIndex_Hmd,
       (vr::ETrackedDeviceProperty)Prop_ArLightDirection_Vector3,
       vr::k_unHmdVector3PropertyTag,
       &m_mainLightDir,
       sizeof(m_mainLightDir),
       &error);

AR Streaming

To stream CloudXR with AR content, the server application, which is an OpenVR app, provides the main scene in the left eye that contains RGBA data. The alpha channel in the data indicates the regions that should be blended with live camera content. The right eye data is ignored, but because OpenVR requires the submission of both eyes, for best performance we recommend that the application also submit the left eye texture as the right eye texture, by using the same texture handle, or optionally submit a small dummy texture for the right eye.

Audio Support

CloudXR supports sending and recieving audio data between the server and client. The Windows and Android Sample clients show how this can be achieved using the CloudXR cxrSendAudio() API and the cxrClientCallbacks.RenderAudio() callback.

Note

Sending audio from the client to server is disabled by default. To enable this the -sa client option and the -ra server option must be set. See Command-Line Options for more information.

Pose to Frame Correlation

CloudXR supports tagging head poses with a user-provided 64-bit unsigned integer value by setting the cxrHmdTrackingFlags_HasPoseID flag and putting the value in the poseID member of cxrHmdTrackingState, and the same value can be seen in the poseID member of the cxrFramesLatched for the frames corresponding to the head pose. This is intended to help client apps identify which exact head pose was used for a rendered server frame.