A video walkthrough is also available for the Sight tool.
When you open your browser at the default link for Sight (http://<ROBOT_IP>:3000/ or
https://<ROBOT_IP>:3000/), you’ll find a page similar to the one below. This is the main Sight window.
The UI is organized into the Channels menu, Windows menu, and Window Container. The left panel can be hidden by clicking “-” and expanded using the box. If you right click the “Isaac Sight v2.6”, a menu appears with the options to “Reconnect” to the backend, “Disconnect” from the backend (and it won’t attempt to reconnect until “Reconnect” is pressed), or “Disconnect Others” which will disconnect all the other connections.
Channels Menu
The Channels menu (bottom left) lists all of the channels the application is publishing to. The channels are sorted first by application, then node, then component; the name of the channel is split using “.”. Internal nodes with only one child are merged with their child to reduce the size of the tree.
Internal nodes are easily recognizable by the adjacent arrow showing the current status, whether or not their children are visible.
When a channel is disabled from the Channels menu, no data is sent from the server. This saves bandwidth and computation load on the application, but the information is not logged or otherwise preserved for later analysis.
You can enable or disable any section of the tree with a single click.
You can also right-click a portion of the tree to either create a new 2D/3D Rendering or Plots with all the (compatible) channels of the subtree or add them to an existing renderer.
Windows Menu
The Windows menu (top left) contains a list of all currently available windows, organized into four categories:
2D Drawing: A window showing renderings of 2D operations such as drawn images, drawn lines, and circles. They are customizable and can be hidden using the checkbox or deleted using the red “X” button. You can create a new 2D drawing at any time using the button at the top of the menu or by right-clicking the Channel Menu. Refer to 2D Rendering for details.
3D Rendering: Similar to 2D renderings, but instead drawn in 3D using WebGL. Refer to 3D Rendering for details.
Plots: A list of windows showing plots and times series. See Plots for details.
Others: A list of all other windows, such as the posetree and config.
Every window can be hidden using its corresponding checkbox. Clicking the name of a window scrolls the view to that window.
Window Container
The Window Container contains all windows currently visible and listed in the window menu. Windows can be dragged to different locations to customize the view or hidden by clicking the top-right “-” icon. To display a window that has been hidden, click the check box for the window in the Windows menu
List of Window Types
Sight has the following window types, described in the following sections:
2D Rendering: 2D rendering windows show a customizable 2D view of the robot. They can be used to display camera images and maps with overlays such as scan beams, robot position, detected obstacles, and others.
3D Rendering: 3D rendering windows are similar to 2D drawing windows except that they are rendered in 3D.
Plot: Plot windows display graphical representations of information that is useful for system monitoring.
Widget: Some examples of existing widgets will be presented later; more can easily be created and added.
Shortcuts
The following keyboard shortcuts are available:
F9: Press enter to reorganize the windows so that they don’t overlap.
Right-click: Right-click on a rendering to display the available menu.
Middle-mouse click: While pointing to a channel name on the left menu, click the middle button of the mouse to copy the full name to the clipboard.
You can create a new 2D or 3D rendering using the Add 2D or Add 3D button in the Windows menu. Once the window is created, you can choose what channel to render and in which order. You can also create a new rendering by right-clicking a channel in the Channels menu and selecting Create a new 2D Renderer or Create a new 3D Renderer.
Right-click rendering window to access the quick menu:
Duplicate Renderer: Create a copy of the renderer; all channels and settings are transferred to the new renderer. This can be used to quickly change settings while keeping the old renderer unchanged.
Enable all channels: Enable in the Channels menu all the channels present in this renderer.
Check all channels/Uncheck all channels: Check or Uncheck the channels in the legend of the renderer. If a channel is not checked, it won’t be rendered.
Channel display: * Hide Channels: Hide the legend that contains the list of channel from the view. * Append Channels: Append the legend below the view * Sticky bottom: If the view is quite larger than the window, the legend might not be visible. This option will keep the legend at the bottom of the window even if it overlaps with the view.
Save canvas: Take a screenshot of the current view
Start and stop recordings: Start and stop a recording in “.webm”.
Window Fit Content: Match the size of window to the size of the view (plus the size of the legend if visible). The window will not resizable from the edge.
Allow resizing window: The alternative to the Window Fit Content option; the window can be resized by dragging the edges of the window.
Settings: Access the settings of the rendering.
Change delay: By default, the renderer displays with a 0.2s delay, which allows it to receive and synchronize all messages. The delay can be set from from 2.0s to 0s.
Use estimated time: Time moves forward based on the latest time received by the PoseTree and the time since it was received . The downside of this method is that, if the latency of the network is not constant, the time might jump backward.
Use latest time: Time is provided by the PoseTree (i.e. the time of the latest update).
Image orientation (2D renderer only): Rotate the view by 0/90/180/270 degrees.
Show pixel information/Hide pixel information (2D renderer only): Show/hide the pixel and color of the current pixel pointed at by the mouse.
Show current time as default (Plot only): Show the current value of the plots; furthermore, a white line is displayed, representing the current time in the plot.
There is a channel list below the view in the legend. Clicking on the icon in the left side of a given channel gives access to channel specific menus. Use the checkbox to enable or disable a channel from the view.
Channels that appear red are not enabled in the Channels Menu and are therefore not being updated. Channels in yellow are not rendered (hover over a channel to get more information); in general, this is because the main frame of the channel is not connected to the rendering frame of the renderer.
If the messages arrive at a renderer with a higher latency than the default 0.2s delay (see the Change delay option above), or if some messages are dropped (either due to throttling from the codelet or from the network itself), this might also result in rendering channels out of sync.
2D Rendering
2D objects are rendered layer-per-layer and in the order they are listed. Each channel comprises one layer, and the layer menu lets you pick the transparency (alpha channel), the size (for points and lines), and the default color (if none is specified). In addition, for each channel, you can select a time window to use for rendering; if the time window is greater than zero, then all the messages that fall inside the window will be rendered.
3D objects are rendered using an orthographic projection along the Z axis for map view, and a perspective projection using the provided pinhole model if provided.
If an image is rendered in the first channel, by default the window is automatically resized to match the size of the image (this behavior can be changed using the Allow resizing window option in the quick menu).
The mouse wheel can be used to zoom in and out of the produced image. It can also be used to scroll the view by clicking and dragging the image.
The Settings menu can be accessed by right-clicking the rendering window and selecting Settings. It contains the following:
The window name
The current dimensions of the rendering (these can be changed).
The rendering frame information: * Static frame: The frame used for time synchronization (in general, it should be a frame considered static in the world) * Rendering frame: The frame used for rendering * Scale: The size of a pixel * Transform: A constant transform (X, Y, Alpha) added after the rendering frame.
High Quality Zoom: If selected, the rendering occurs directly on the final canvas (zoom applied); otherwise, the rendering occurs on the renderer dimension and then scaled up/down.
Background color: The color used to clear the background. Most of the time, there is a background image used as first channel; otherwise, the background color will be used.
A drop-in menu with the list of all available channels. You can select a new channel and press the Add channel button.
A drop-in menu with the list of all available markers. You can select a new marker and press the Add marker button.
A list of all channels currently being rendered. On the left side, the arrows let you reorder channels. Use the icon on the right to remove channels. You can add a new channel directly from the channel menu by right-clicking the channel name. You can also create a new renderer directly from a channel or group of channels using the Channels menu.
3D Rendering
For each layer of the a 3D rendering, the color, size, and transparancy can be customized similar to a 2D rendering, and a default Z order can be set for rendering 2D objects in the 3D world. You can create walls out of images for a binary map. In addition, for each channel, you can select a time window to use for rendering; if the time window is greater than zero, then all the messages that fall inside the window will be rendered.
The view can be controlled using the mouse or keyboard:
Translation: Left click plus keyboard: * Arrow up or “W”: forward * Arrow down or “S”: backward * Arrow left or “A”: left * Arrow right or “D”: right * “Q” or “R”: up * “E” or “F”: down
Rotation: Hold the mouse left-click while dragging the mouse.
Enter or leave fullscreen mode: Double left-click the mouse.
Exit fullscreen mode: Press the “Escape” key
The Settings menu can be accessed by right-clicking the rendering window and selecting Settings. It contains the following:
The window name
The current dimensions of the rendering (these can be changed).
The rendering frame information: * Static frame: The frame used for time synchronization (in general, it should be a frame considered static in the world) * Rendering frame: The frame used for rendering
Background color: The color used to clear the background. Most of the time, there is a background image used as first channel; otherwise, the background color will be used.
Use latest mesh: When this option is enabled, the mesh operation will be constructed in a more efficient way and rendering will be faster; however, Sight will always use the latest mesh instead of the mesh at the rendering time.
A drop-in menu with the list of all available channels. You can select a new channel and press the Add channel button.
A drop-in menu with the list of all available markers. You can select a new marker and press the Add marker button.
A list of all channels currently being rendered. On the left side, the arrows let you reorder channels. Use the icon on the right to remove channels. You can add a new channel directly from the channel menu by right-clicking the channel name. You can also create a new renderer directly from a channel or group of channels using the Channels menu.
Plots can be created with channel streaming values or time series:
Stream values are plotted one point per value at the provided time (points are linked together using a line segment) time series are plotted. Values plotting is good for monitoring some scalar-like dynamics (speed, acceleration, etc.), delay, scores, etc.
Time series are plotted on at time (based on the current time of the renderer) as a full plot. Time series are good for visualizing trajectories or histograms that change over time.
For each channel, you can select the color in the bottom legend.
The view can be controlled using the mouse:
Rescale using the mouse wheel (change the delta time of a pixel).
Move the view forward/backward by clicking and dragging with the mouse wheel.
The settings menu contains the following:
The current dimensions of the rendering (these can be changed).
Decimal Precision: The number of decimal digits used to print the value
Y Axis: By default, the Y axis scales automatically using the min/max value in the range, but it can be overridden to a fixed value if needed.
A tree showing the list of all the channels that contain a plot or time series; enable or disable some of the channels to add them or remove them from the renderer.
The plot history is controlled by the PoseTree widget (described below), which has a configurable
history parameter. Plots will be kept up to a certain amount of data, but older data will be
dropped after the limit is reached. If you notice plot data being dropped, you can increase the
maximum amount of data allocated to the history.
The PoseTree widget must be present for the renderer to work properly. It contains a list of frames in the application and describes how they are connected to each other. It also provides access to the transformation.
Control:
Frames can be moved around by click and dragging them.
Zoom in and out using the mouse wheel.
Translate the view by left-clicking and dragging it.
Left-click two frames to get the current transformation between them. For 2D pose, the X, Y, and Angle are provided. For 3D pose, the X, Y, Z, and Quaternion (w, x, y, z) are provided.
Right-click a frame to view a menu on the right with the list of edges. The subtree of each edge can be hidden or shown.
The Settings menu on the bottom controls the rendering time of the application. It also allows logging and replay of the data that the frontend has received:
Time offset: Allows you to go back in time or add a constant delay to all renderers.
History: The minimum history that the channels keep. You can click the number to edit the value.
Current time: The number of seconds since the application started.
Navigation buttons: * <: move the time offset by 0.05 into the past. * Replay/Stop: Reduce the offset until it reaches 0s. * >: Move the time offset by 0.05 into the future.
Save logs: Save the log (in JSON format) containing all information about the frontend.
Load logs: Load a previously saved log and replay it. This option is only available when the application stops.
The Sight logs recorder contains the configuration of the application, the setup of the frontend, and data that was sent to sight in the period provided, both for plots and rendering. The logs are stored as JSON to make them human readable, which can be convenient for debugging; however, the file size is often quite large, so if you need to share them, consider compressing them first.
The Config widget contains the list of the entities of the application, the list of components, information about the type, the list of parameters, and the current value of the parameters. You can also use it to modify the parameters.
To modify a parameter, you need to first identify the node it belongs to and search for it in the list (you can use the filter at the top to help), then select the component, and finally click on the value of the parameter you wish to change. An edit box will appear: you must enter a valid, parsable YAML string. To cancel the edit, press Escape; to finish the edit, either click outside the text area or press Enter. An Update button should appear on top. Once you’ve made all updates, press the Update button to send them to the backend. If you want to undo some changes, press the red “X” next the parameter that was modified, or press Refresh to undo all the changes and get the latest value of the backend.
Channel statistics are available for all applications. You can display it by clicking Channels in the Channel menu. It contains a list of channels with various statistics:
Node: The name of the entity that the message comes from
Codelet: The name of the codelet that the message comes from
Channel: The name of the channel
Frequency: The number of messages the channel receives per second (moving average)
Bandwidth: The approximate bandwidth for sending uncompressed messages. Real bandwidth might be lower if the server compresses the messages.
Frame rate: The approximate frame rate of the renderer
Data size: The average size of a message
Latency: The average delay between receiving the message and the timestmap the message contains
Type: Can be one of the following: “Image”/”Plot”/”2D”/”3D”/”Text”/”Isaac”. “Isaac” is used for widgets messages.
On top of the widget, there is a small plot indicating the total bandwidth of the system as well as the latency and the frame rate. The latency is computed using a system of “ping”/”pong” messages, while the frame rate is estimated over the last second.
To create a custom widget, you need to provide a new JavaScript file:
The following code creates a new window:
// Config options:
// - resize: whether resize the object when the window is resized
// - onrsize: callback called when resized
// - close: Whether the window can be destroyed (default is false)
// - onclose: callback when the window is closed
// - onmaximize: callback when the window is not hidden anymore
// - width/height: dimensions of the window.
// - menu_label: Label of the menu.
// - hide: Hide the window at creation
createWindow(obj, name, options = {}) {
WindowManager().createWindow(widget_div, "Widget Name", /*config*/ {
hide: true,
resize: false
});
You can register a callback for receiving some messages with a specific type:
// Registers the callback
application_manager_.registerMessageCallback("message_type", function(message, size, socket) {
statistics_recorder_.addMessage("///Config", size, "Isaac"); // To record for the channel stats
// do something with the messages
});
application_manager_.addAcceptedMessage("message_type");
You can register a callback when a new connection is made or closed:
application_manager_.registerOnOpenCallback(function() {
// For example create the window.
});
application_manager_.registerOnClostCallback(function() {
// For example destroy the window.
});
To send a message to the backend, use the following:
let message = JSON.stringify({type:"message_header", data: { request: "get" }});
my_socket.send(message);
// If more than one application is connected and you want to send them a message:
for (let i = 0; i < my_socket.sockets.length; i++) {
// If the socket is started, we can send the request, otherwise we add it to the list.
if (my_socket.sockets[i].readyState == 1) {
my_socket.sockets[i].send(message);
} else {
my_socket.sockets[i].to_send.push(message);
}
}
To have your JavasScript/CSS files loaded by the frontend, you can use either the JSON config or register the files inside a SightReceiver.