Remote Joystick using Sight

The conventional way of controlling the movement of a robot is by using a joystick directly connected to the robot. However, in Isaac, the Virtual Gamepad Sight widget can be used to send signals simulating traditional joystick axes to a robot, over the network. The widget allows three input mechanisms, of which only one can be used at a time:

  • A browser recognized standard gamepad controller connected to the machine running Sight Client, which makes use of the browser’s Gamepad API.
  • A mouse connected to the machine running Sight Client. The mouse interacts with the virtual mousepad in the widget, that simulates joystick axes.
  • A keyboard connected to the machine running Sight client. Directional keys are assigned on the keyboard to simulate joystick axes.

To use the Virtual Gamepad Widget from Sight, set up VirtualGamepadBridge in the application’s JSON files as follows:

  1. Add the following node:

{ "name": "virtual_gamepad_bridge", "components": [ { "name": "message_ledger", "type": "isaac::alice::MessageLedger" }, { "name": "VirtualGamepadBridge", "type": "isaac::navigation::VirtualGamepadBridge" } ] }

  1. Add the following configuration parameters to the node:

"virtual_gamepad_bridge": { "VirtualGamepadBridge": { "tick_period": "100ms" } }

  1. Add connections to enable communication between Sight and the backend:

{ "source": "websight/WebsightServer/virtual_gamepad", "target": "virtual_gamepad_bridge/VirtualGamepadBridge/request" }, { "source": "virtual_gamepad_bridge/VirtualGamepadBridge/reply", "target": "websight/WebsightServer/virtual_gamepad_reply" }

  1. Add connection to the codelet processing Joystick-kind messages (for example: RobotRemoteControl):

{ "source": "virtual_gamepad_bridge/VirtualGamepadBridge/joystick", "target": "carter_joystick/isaac.navigation.RobotRemoteControl/js_state" }

  1. Include the navigation module in your application BUILD file:

isaac_app( name = "...", ... modules = ["navigation"], )

  1. Include the navigation module in your application json file:

modules = ["navigation"]

VirtualGamepadBridge has the following configuration parameters:

  • tick_period: interval between periodic tick() function calls.
  • sight_widget_connection_timeout: Time in seconds to wait before automatically disconnecting from the Sight Virtual Gamepad Widget instance. Default is 30 seconds.
  • num_virtual_buttons: Number of buttons on a simulated Joystick that the front-end mousepad or keypad are simulating. The default is 12.
  • deadman_button: Button number for failsafe button. This button number is used in the messages generated from the mousepad or keypad simulating a joystick. The default is 4.

The following is the Virtual Gamepad Widget in Sight:


Elements of the Widget

  • Widget ID: A unique ID per widget instance, used to make sure that only one such instance of the widget is connected to the backend, so that only one user at a time is in control.
  • Connect to Backend button: Click to make a live and unique connection with the backend. The current connection status shows as green or red icon on the button. Upon successful connection, the modes panel is displayed in the center. If the widget remains disconnected, the modes panel remains hidden to disallow user interaction.
  • Mode Selector Buttons: Click to select a mode of operation. The selected/visible mode activates to be the current mode of operation. The button for the selected mode switches from grey to green.
  • Virtual Mousepad Joystick dial: Use a connected mouse or touchpad to interact with the dial using the cursor to send simulated joystick signals to the backend. See Modes of Operation for more information.
  • Virtual Keypad Joystick presentation: When enabled, displays the current key being pressed on the connected keyboard. This is only a visual presentation of the key being pressed. These are not buttons that can be clicked with a mouse or touchpad. See Modes of Operation for more information.
  • Backend Connected to Widget display: Shows the ID of the widget instance currently connected to the backend, as an indication of which user is in control.
  • Connected Standard/Remapped Gamepads display: Shows the number of gamepads currently recognized by the browser running Sight. This widget allows control using a connected controller only if a single controller is recognized by the browser. If multiple controls are connected, all but one must be disconnected.

If the procedure in Setting up an Isaac Application to use the Virtual Gamepad Widget is not completed, no VirtualGamepadBridge codelet is instantiated in the backend when launching an application. In such a scenario the Sight widget remains disabled with the following message:


When setup is complete, the Virtual Gamepad Widget can be used after the following steps are performed:

  1. Connect to the backend by clicking the Connect to Backend button.
  2. Select the desired mode by clicking the corresponding mode button. The status of a mode is indicated by the color of its button. See Modes of Operation for explanation of the modes.
  3. To disconnect your instance of the widget from the backend, click the same button used before for connection.

Only one instance of such a widget can remain connected to the backend. This prevents multiple Sight users with multiple Sight instances from attempting to move the same robot at the same time.

Modes of Operation

Gamepad Mode


In Gamepad mode, a controller or gamepad, connected either wirelessly or via USB to the machine running Sight and recognizable by the browser’s Gamepad API controls the robot. Note that only one standard controller should be connected under this mode of operation. If only one standard controller remains connected, the gamepad icon turns green, else the icon color remains grey representing that no controller is usable. The gamepad functions as though it is directly connected to the robot.

Mousepad Mode


To use the a connected mouse or a touchpad with the mousepad dial, left click and hold on the small dark colored circle in the center. While keeping the left click pressed, move the cursor to move the dark circle in the bigger circular dial.

When the dark circle is in the green zone on the dial, the widget sends simulated joystick messages to the backend. If it is in the grey zone, no messages are sent to the backend.

The dark circle can be moved with only left click pressed, else it returns to its center position on the dial, indicating no joystick activity.

If the cursor moves out of the dial radius, the dark circle resets itself to the center position.

The closer the dark circle is to the periphery, the closer a simulated joystick axis value is to its extreme. For example, for slower speeds, the dark circle needs to remain closer to the center of the dial, within the green zone. For higher speeds, the dark circle must be close to the periphery.

Keypad Mode


The W, S, A, D keys, or any combination of those, on a connected keyboard can be used to send simulated joystick axes messages to the robot.

The UI indicates when a directional key is pressed or released.

© Copyright 2018-2020, NVIDIA Corporation. Last updated on Oct 30, 2023.