ARCore Client (Android)

The NVIDIA CloudXR SDK includes a sample ARCore client for AR streaming, which is designed to work with Android tablets and phones that support the ARCore SDK. The client decodes the video and alpha content that is streamed from the CloudXR server and blends it with the Android device’s camera view. The client collects motion data from the Android device and sends it to the CloudXR server.

The ARCore client is supplied as source code and as a prebuilt .apk file. The .apk file is in the {sdk-root-folder}\Sample\Android\GoogleAR folder.

The Android device must be capable of decoding 4k HEVC video @ 60fps. The provided sample client has been tested with the Samsung Galaxy S10+ device.

ARCore Client Prerequisites

Required

  • An Android tablet or phone that supports the ARCore SDK, for example:

    • Samsung Galaxy S21

    • Google Pixel 5

  • Android Debug Bridge (ADB) to install the client application without needing full developer tools.

Optional

For developers who want to modify, build, and debug the client application, a complete Android development enviroment is required:

  • Android Studio 4.0 or later.

  • Android SDK 7.1.1 ‘Nougat’ (API level 25) or higher.

  • Android build tools 28.0.3

  • Android NDK 21.4.7075529

  • OpenJDK 1.8n

Note

  • Android Studio can be downloaded from https://developer.android.com/studio.

  • Installing Android Studio also installs Android Platform Tools, Android SDK, and OpenJDK.

  • To install Android NDK from the Android Studio menus, click Tools>SDK Manager, select SDK Tools and then NDK, and click OK.

  • To use the bundled OpenJDK, click File>Project Structure in the menu bar. In the SDK Location page under JDK Location, select the Use embedded JDK checkbox, and then click OK.

  • The NDK version was selected because it was the LTS release at time of development and is aligned with Android Studio 4.0.x as the default NDK version.

    Newer NDK versions, such as 23.x, might cause build issues.

  • As of the CloudXR 3.1 release, only Android 64-bit builds are supported.

You also need:

Note

The gradle project has ARCore as a dependency, so the project will automatically download and cache the SDK. However, you might need to adjust the dependent version number based on the devices that you support. The sample is currently based on ARCore SDK 1.26.0.

Building the ARCore Client

Note

If you plan to use the prebuilt .apk file you can proceed to Installing the ARCore Client.

  1. Go to https://github.com/NVIDIA/CloudXR/tree/master/arcore-android-sdk/samples/hello_cloudxr_c and download the entire hello_cloudxr_c project and place it in the {sdk-root-folder}\Sample\Android\GoogleAR folder.

  2. Copy Google Oboe SDK .AAR file (oboe-1.5.0.aar) into the {sdk-root-folder}\Sample\Android\OculusVR\hello_cloudxr_c\libs folder.

  3. Copy the CloudXR SDK client package, which is the CloudXR.aar file, from {sdk-root-folder}\Client\Lib\Android folder to the {sdk-root-folder}\Sample\Android\GoogleAR\hello_cloudxr_c\libs folder.

  4. Run Android Studio.

  5. Complete one of the following tasks:

    • Select Open existing Android Studio project on the Welcome screen.

    • Select File > Open.

  6. Navigate to {sdk-root-folder}\Sample\Android\GoogleAR and open the hello_cloudxr_c folder.

  7. Select Build > Make Project.

This process should generate an .apk file in the {sdk-root-folder}\Sample\Android\GoogleAR\hello_cloudxr_c\build\outputs\apk\debug directory that can be used to debug or be installed manually. You can also automatically generate an .apk file by running Android Studio (for the latter, proceed Running the ARCore Client).

Note

If build fails with an error about manifest, in the lower-left side, select File > Settings > Experimental and deselect the Only sync the active variant checkbox.

Note

To build from the command line, run gradlew build from the hello_cloudxr_c folder.

Installing the ARCore Client

Install the .apk file that was created by building the sample application, or the prebuilt .apk, on the Android device.

Note

The pre-built .apk file that is packaged with the SDK is in {sdk-root-folder}\Sample\Android\GoogleAR.

  1. Place the device in developer mode and allow USB connection in debug mode on the device.

  2. Use a USB cable to connect the Android device to the development system.

  3. If prompted on the device to allow connections, select Allow.

  4. In a Command Prompt window, navigate to the folder that contains the .apk file that was created by building the sample application or the pre-built .apk file.

  5. Use ADB to install the application from the release .apk file.

    adb.exe install -r <APK name>.apk

    Note

    By default, the ADB.exe program is installed by Android Studio in C:\Users\{username}\AppData\Local\Android\Sdk\platform-tools

    When the installation is complete, ADB responds with a Success message.

  6. (Optional) Pre-specify the command-line per device:

    1. Create a plain-text file named CloudXRLaunchOptions.txt that contains -s <IP address NVIDIA CloudXR server>.

      For example, for a server with IP = 1.1.1.1, the file should contain -s 1.1.1.1.

    2. Copy the CloudXRLaunchOptions.txt file to the base device folder as shown in Windows Explorer, or if using ADB, to the /sdcard/ directory of the device using the following command:

      adb.exe push CloudXRLaunchOptions.txt /sdcard/CloudXRLaunchOptions.txt

    See Command-Line Options, for more information about using launch options and a complete list of all available options.

Running the ARCore Client

  1. Start SteamVR on the server system.

  2. Start the CloudXR ARCore Client app on your Android device.

  3. When prompted, enter the IP address of the CloudXR server or accept a previously entered value.

  4. Once you see the camera view, you should slowly turn the camera from side to side or up and down until it sufficiently visualizes the target ‘ground plane’ on the screen as a series of connected white dots.

    ../_images/arcore_surface_detect.jpg

    ARCore Surface Detection

  5. Touch the grid to select an anchor point for the AR world.

  6. Wait until the app connects to the server and shows the default SteamVR view (or which application was already running on the server).

    Note

    To stream CloudXR with AR content, the server application, which is an OpenVR app, provides the main scene in the left eye that contains the RGBA data. The alpha channel in the data indicates the regions that should be blended with live camera content.

    The right eye data is ignored, but because OpenVR requires the submission of both eyes, for best performance, we recommend that the application also submit the left eye texture as the right eye texture by using the same texture handle or optionally submit a small dummy texture for the right eye.

    In the {sdk-root-folder}\TestTools folder, a basic server application, ar_test.exe, is provided to help illustrate and validate AR streaming. After the connection is established, start ar_test.exe on the server to see an array of textured cubes blended into the current camera view.

  7. To reset the anchored point in the camera view, press and hold your finger on the screen for a few seconds.

    This step resets the anchored position and allows you to set a new position.

Note

The ARCore sample has extra command-line options that modify how it runs. See ARCore Client Command-Line Options for more information.