Streaming Application Setup for Foveated Streaming#

Foveated Streaming uses a multi-step protocol between the Apple Vision Pro client and the streaming application to discover, authenticate, and establish an XR streaming session.

A complete sample implementation is available on GitHub. To try it out of the box or to study the full implementation details, refer to:

The sample includes a working streaming application with mDNS advertisement, the TCP session management protocol, QR-code pairing, and CloudXR Runtime management via Stream Manager. It can be used as a starting point for building a custom streaming application or as a reference alongside the guidance below.

The rest of this page provides a high-level overview of the architecture and responsibilities for developers who want to integrate the streaming logic into their own application.

Architecture Overview#

When a Foveated Streaming client uses the .systemDiscovered endpoint, the connection goes through a discovery and session management protocol before CloudXR streaming begins. Your streaming application is responsible for implementing this protocol alongside managing the CloudXR Runtime via Stream Manager.

The connection flow involves three layers of communication between the client and the streaming application:

  1. mDNS Service Discovery (optional): The streaming application advertises itself on the local network so the Apple Vision Pro can discover it automatically. If mDNS is not used, the client can connect directly by specifying the IP address and port.

  2. TCP Session Management Protocol: The streaming application implements a length-prefixed JSON protocol over TCP that handles connection negotiation, authentication (QR-code pairing), and session lifecycle events.

  3. CloudXR Streaming: The CloudXR Runtime handles the actual XR content streaming once the session management protocol completes.

Step 1: mDNS Service Advertisement#

Your streaming application can optionally advertise itself on the local network using mDNS (Bonjour/Zeroconf) so that Apple Vision Pro devices can discover it automatically. If mDNS is not used, the client can connect directly using the IP address and port via the .local endpoint.

Service Configuration#

Advertise a service with the following parameters:

Parameter

Value

Service Type

_apple-foveated-streaming._tcp

Instance Name

The machine’s hostname (or a custom display name)

Port

The TCP port for session management (e.g., 55000)

TXT Record

Application-Identifier=<your.app.bundle.id>

The Application-Identifier in the TXT record must match the bundle identifier of your visionOS client application. This allows the Vision Pro to filter and discover only the streaming endpoints that are compatible with your app.

Step 2: TCP Session Management Protocol#

After the client discovers the streaming application via mDNS, it establishes a TCP connection and begins a JSON-based session management protocol. This protocol handles connection negotiation, QR-code pairing authentication, and session lifecycle events.

The full protocol specification, including message format, message types, and expected behavior for each message, is defined in the Foveated Streaming sample project.

At a high level, the streaming application must handle the following message flow:

  1. RequestConnection: The client initiates a session. The streaming application responds with its identity and certificate fingerprint (AcknowledgeConnection).

  2. RequestBarcodePresentation: If the client has not previously paired with this streaming endpoint, it requests QR-code authentication. The streaming application generates and displays a QR code, then confirms with AcknowledgeBarcodePresentation.

  3. SessionStatusDidChange: The client reports session state transitions (WAITING, CONNECTING, CONNECTED, PAUSED, DISCONNECTED). On WAITING, the streaming application starts the CloudXR Runtime and sends MediaStreamIsReady. On DISCONNECTED, it stops the runtime and cleans up.

  4. RequestSessionDisconnect: The streaming application can send this to request the client to end the session.

Step 3: QR-Code Pairing#

QR-code pairing is used for first-time connections or when the certificate fingerprint has changed. The QR code contains authentication credentials that the Apple Vision Pro scans to establish trust with the streaming application.

Generating the QR Code Payload#

The QR code payload is a JSON string containing two values obtained from the Stream Manager:

{
    "token": "<client-token>",
    "digest": "<certificate-fingerprint>"
}

To generate these values:

  1. Client Token: Call SetClientId on the Stream Manager with the ClientID received from the RequestConnection message. The Stream Manager returns an authentication token.

  2. Certificate Fingerprint: Call GetCryptoKeyFingerprint with SHA-256 algorithm on the Stream Manager.

// Using the Stream Manager C API (NvStreamManagerClient.h)

// 1. Generate client token from the ClientID
char token[256];
size_t token_size;
nv_rpc_client_set_client_id(client, clientID, strlen(clientID),
                             token, sizeof(token), &token_size);

// 2. Get the certificate fingerprint
char fingerprint[256];
nv_rpc_client_get_crypto_key_fingerprint(client,
    NV_CRYPTO_ALG_SHA256, fingerprint, sizeof(fingerprint));

// 3. Compose the QR code payload
// JSON: {"token": "<token>", "digest": "<fingerprint>"}

Encode this JSON string into a QR-code image and display it on screen. The user scans this QR code with the Apple Vision Pro to complete pairing.

Note

After a successful QR-code pairing, the Apple Vision Pro stores the authentication token and associated certificate fingerprint. Subsequent connections to the same streaming endpoint skip QR-code pairing as long as they have not changed.

Step 4: CloudXR Runtime Management#

Your streaming application must manage the CloudXR Runtime lifecycle using the Stream Manager. The Stream Manager provides an RPC interface for starting, stopping, and monitoring CloudXR services.

Launching the Stream Manager#

Your application must launch NvStreamManager.exe as a subprocess when it starts:

// Launch NvStreamManager.exe from the Server/ subdirectory
// It should run as a background process with no console window.
// The process should be terminated when your application exits.

Connecting to the Stream Manager#

After launching, connect to the Stream Manager via its named pipe RPC interface:

#include "NvStreamManagerClient.h"

nv_rpc_client_t client;
nv_rpc_client_create(NULL, &client);  // Default pipe name
nv_rpc_client_connect(client);

Starting CloudXR on Session WAITING#

Start the CloudXR service and notify the client:

nv_rpc_client_start_cxr_service(client, "6.0.0", 5);

Stopping CloudXR on Disconnect#

When the client sends SessionStatusDidChange with status DISCONNECTED, stop the CloudXR service:

nv_rpc_client_stop_cxr_service(client);

For the complete Stream Manager API reference, refer to Stream Manager.

Application Packaging#

Important

Both the CloudXR Runtime and Stream Manager must be shipped with your streaming application. They are not installed system-wide and must be colocated with your application binary.

Place the runtime files inside the Server/releases/<version>/ folder structure. Refer to Stream Manager for details on the Stream Manager’s expected folder layout.

Additional Resources#