ARCore Client (Android)

The NVIDIA CloudXR SDK includes a sample ARCore client for AR streaming, which is designed to work with Android tablets and phones that support the ARCore SDK. The client collects motion data from the Android device and sends it to the CloudXR server as ‘pose’ data. The server renders scene frames based on the client position, and streams them to the client. The client decodes the video (and alpha when provided) streamed from the CloudXR server and blends it with the Android device’s camera view.

Note

Note that there is nothing preventing a non-AR scene, call it “2D-VR”, where you are in a 3D space, you can ‘look around’ with the camera, but moving/translocating in the space would require some custom coding for user input.

This sample was started based on the hello_ar_c sample application from the ARCore SDK.

ARCore Client Prerequisites

Required

  • An Android tablet or phone that supports the ARCore SDK and is capable of 4K HEVC decoding at 60fps or higher. Some example devices include:

    • Samsung Galaxy S21, S22, S23

    • Google Pixel 5, 6A, 7, 8

  • Android Debug Bridge (ADB) to install the client application without needing full developer tools.

Optional

For developers who want to modify, build, and debug the client application, a complete Android development enviroment is required:

  • Android Studio 4.0 or later.

  • Android SDK 7.1.1 ‘Nougat’ (API level 25) or higher.

  • Android build tools 28.0.3

  • Android NDK 21.4.7075529

  • OpenJDK 1.8n

Note

  • Android Studio can be downloaded from https://developer.android.com/studio.

  • Installing Android Studio also installs Android Platform Tools, Android SDK, and OpenJDK.

  • To install Android NDK from the Android Studio menus, click Tools>SDK Manager, select SDK Tools and then NDK, and click OK.

  • To use the bundled OpenJDK, click File>Project Structure in the menu bar. In the SDK Location page under JDK Location, select the Use embedded JDK checkbox, and then click OK.

  • The NDK version was selected because it was the LTS release at time of development and is aligned with Android Studio 4.0.x as the default NDK version.

    Newer NDK versions, such as 23.x, might cause build issues.

  • As of the CloudXR 3.1 release, only Android 64-bit builds are supported.

You also need:

Note

The gradle project has ARCore as a dependency, so the project will automatically download and cache the SDK headers and libraries. However, you might need to adjust the dependent version number based on the devices that you need to support, and the manual SDK you download.

Building the ARCore Client

In the below instructions {cxr-root} is the path to the CloudXR SDK.

Note

On Windows you may need to move the SDK to the top level of your drive due to path length restrictions. You’ll get strange errors from the build process about files not existing, but will also note that the paths being shown are extremely long.

  1. Unzip the ARCore SDK into the {cxr-root}\Sample\Android\GoogleAR folder, resulting in the contents being in a subfolder like {cxr-root}\Sample\Android\GoogleAR\arcore-android-sdk-1.26.0. To simplify next steps, we will use {arcore-root} to refer to that directory.

  2. Go to https://github.com/NVIDIA/CloudXR/tree/master/arcore-android-sdk/samples/hello_cloudxr_c and download the entire hello_cloudxr_c project and place it in {arcore-root}\samples, at the same level as the ARCore SDK samples.

  3. Copy the Google Oboe SDK .AAR file (oboe-1.5.0.aar) into the {arcore-root}\samples\hello_cloudxr_c\libs folder.

  4. Copy the CloudXR SDK client package, which is the CloudXR.aar file, from {cxr-root}\Client\Lib\Android to the {arcore-root}\samples\hello_cloudxr_c\libs.

  5. Run Android Studio.

  6. Complete one of the following tasks:

    • Select Open existing Android Studio project on the Welcome screen.

    • Click File > Open.

  7. Navigate to {arcore-root}\samples and select the hello_cloudxr_c project/folder to open.

  8. Select Build > Make Project.

This process should generate an .apk file in the {arcore-root}\samples\hello_cloudxr_c\build\outputs\apk\debug directory that can be used to debug or be installed manually.

Note

If build fails with an error about manifest, in the lower-left side, select File > Settings > Experimental and deselect the Only sync the active variant checkbox.

Note

To build from the command line, run gradlew build from a command prompt in the hello_cloudxr_c folder.

  1. Prepare the device for running/debugging

    1. Place the device in developer mode and allow USB connection in debug mode on the device.

    2. Use a USB cable to connect the Android device to the development system.

    3. If prompted on the device to allow connections, select Allow.

Installing the ARCore Client

Note

This section is only necessary should you want to manually install from command-line. If running through Android Studio, it will take care of the installation, so you can skip ahead to Running the ARCore Client.

  1. In a Command Prompt window, navigate to the folder that contains the .apk file that was created by building the sample application.

  2. Use ADB to install the application from the release .apk file.

    adb.exe install -r <APK name>.apk

    Note

    By default, the ADB.exe program is installed by Android Studio in C:\Users\{username}\AppData\Local\Android\Sdk\platform-tools

    When the installation is complete, ADB responds with a Success message.

  3. If you are building from the command-line with Gradle, it can do the installation: gradlew installRelease

Running the ARCore Client

Note

See Command-Line Options, for more information about our command-line/launch options system, and a complete list of all the available options, and How to Launch on Android for more detailed instructions on the various methods for starting Android applications.

  1. Start SteamVR or the CloudXR Experimental Server Sample on the server system.

  2. Launch the CloudXR ARCore Client app on your Android device (via whichever method you choose).

  3. When prompted, enter the IP address of the CloudXR server or accept a previously entered value.

  4. Once you see the camera view, you should slowly turn the camera from side to side or up and down until it sufficiently visualizes the target ‘ground plane’ on the screen as a series of connected white dots.

    ../_images/arcore_surface_detect.jpg

    ARCore Surface Detection

  5. Touch the grid to select an anchor point for the AR world.

  6. Wait until the app connects to the server, and starts streaming.

  7. For SteamVR, you will get whatever is configured as your home view, or whatever VR application was already running on the server.

    Note

    To stream CloudXR with AR content, the server application provides the main scene in the left eye that contains the RGBA data. The alpha channel in the data indicates the regions that should be blended with live camera content.

    The right eye data is ignored, but because OpenVR requires the submission of both eyes we recommend that the application also submit the left eye texture as the right eye texture by using the same texture handle, or optionally submit a small dummy texture for the right eye.

    In the {cxr-root}\TestTools folder, a basic OpenVR server application, ar_test.exe, is provided to help illustrate and validate AR streaming in SteamVR. After the connection is established, start ar_test.exe on the server to see a huge 3D grid of textured cubes blended into the current camera view.

  8. As an alternative to Steam, you can test AR clients with the experimental Server Sample. More details on the Server Sample and using it can be found in CloudXR Experimental Server Sample. After the connection is established, you will see an angel statue in front of you. Note that the scale is currently a bit large, so you may need to physically take a step back. Then you can move the device around to look at the AR model.

  9. To reset the anchored point in the camera view, press and hold your finger on the screen for a few seconds.

    This step resets the anchored position and allows you to scan and set a new root anchor position.

Note

The ARCore sample has extra command-line options beyond all the defaults. See ARCore Client Command-Line Options for more information.