Controller Input and Haptics

CloudXR Framework can forward spatial controller and stylus input from visionOS clients to the CloudXR Runtime. Compatible controllers and styli can also receive haptic feedback from the server application.

Controller Setup

Apps that use spatial controller or stylus tracking must declare accessory tracking usage in Info.plist. Apps should also declare the GameController profiles they support. Include SpatialGamepad for spatial controllers, and include other profiles only if your app supports them.

<key>NSAccessoryTrackingUsageDescription</key>
<string>This app uses controller tracking during CloudXR streaming.</string>
<key>GCSupportedGameControllers</key>
<array>
    <dict>
        <key>ProfileName</key>
        <string>SpatialGamepad</string>
    </dict>
    <dict>
        <key>ProfileName</key>
        <string>ExtendedGamepad</string>
    </dict>
</array>

For spatial controller and stylus tracking, configure controllerTrackingMode. If your RealityKit view also receives GameController events, use handlesGameControllerEvents. When that view handles spatial stylus events, include .stylus in the matching options on visionOS 26.2 or later:

var config = CloudXRKit.Config()
config.controllerTrackingMode = .spatialController

RealityView { content in
    // Add CloudXR session content.
}
.handlesGameControllerEvents(
    matching: [.gamepad, .stylus],
    withOptions: .receivesEventsInView(false)
)

Use .spatialController for both supported spatial controllers and the Logitech Muse stylus. The Framework registers the Muse stylus with the logitech/mx_ink_stylus_logitech profile and non-handed chirality. During tracking, CloudXR Framework equips the same stylus controller instance into the left or right hand slot based on the held hand reported by ARKit accessory tracking.

CloudXR Framework maps Logitech Muse stylus input to tip force, middle-button force, front-button click, grip pose, aim pose, and tip pose. You can tune stylus pose prediction separately with stylusTrackingPredictionFactor.

The runtime presents controller and stylus input to the server application through the standard OpenXR action system. For runtime-side controller profiles and binding fallbacks, refer to Controller and Hand Tracking.

Controller Haptics

CloudXR Framework forwards controller haptic requests from the runtime to compatible physical controllers and styli. On visionOS, haptics require visionOS 26.0 or later and a device that exposes GameController haptics, such as PlayStation VR2 Sense controllers or a Logitech Muse stylus.

Your OpenXR server application can use standard OpenXR haptic actions. Haptic requests are delivered when the connected client and device support them. If haptics are unavailable, the request is ignored, but controller or stylus input continues to work.