Questions, Comments, and Bugs

To contact NVIDIA with questions and comments about NVIDIA CloudXR SDK or to report a bug against the SDK, go to the CloudXR Forum.

If you are reporting a bug, provide the following information in your e-mail:

  • Operating system.

  • GPU model.

  • NVIDIA graphics driver version.

  • Headsest/Device used.

  • Steps to reproduce the issue.

  • Log files with verbose mode enabled from the client and server.

    See Log Files for more information.

  • For performance or network issues, share ETL files.

    See ETL Traces for more information.

Log Files

Windows

On Windows, the client and server create log files in the C:\Users\*user-name*\AppData\Local\NVIDIA\CloudXR\logs\ folder.

Additional messages are sent to the debug output, which can be captured by a standard debugger. Additionally, for the Windows client, the following command line arguments can offer additional help:

  • -w: Windowed mode.
    Instead of driving a VR headset, this will display the video feed to the PC screen.
  • -v: Enables verbose logging.
    When sending logs to NVIDIA for debugging, ensure that verbose logs are captured by using this argument.

If the server crashes, the crashdump.dmp file, is created in the log directory.

Android

On Android client devices, the logs are in the /sdcard/CloudXR/ folder.

To capture more detailed log messages from a headset for an Android client, connect the headset to a windows PC and run the adb.exe logcat CloudXR \*:S command.

Log File Names

  • CloudXR Server log file: CloudXR Server log <Date Time stamp>.txt

  • CloudXR Client log file: CloudXR Client Log <Date Time stamp>.txt

  • Stream Server log file: Streamer Server Log <Date Time stamp>.txt

  • Stream Windows client log file: Streamer Client Log <Date Time stamp>.txt

Note

To prevent storage space overrun, the log files on both client and server are capped at 4MB. If the maximum value is reached, a ---- LOG TRIMMED ---- message is displayed in the log file, and log files older than 3 days are deleted. If the files need to be retained for longer than this period, copy them to another location.

Additional Logging Features

Frame timing information will show up in the CloudXR Client Log when running in -v verbose mode, for all client platforms. The timing output is in the following format:

[16:33:59] fps: 71.7, s2st:  30.3, qt:  7.5, late:   0.0 ms
[16:33:59] fps: 71.8, s2st:  28.8, qt: 10.0, late:   0.0 ms
[16:33:59] fps: 71.9, s2st:  28.3, qt: 11.4, late:   0.0 ms
[16:33:59] fps: 12.1, s2st:  27.7, qt:  8.2, late:  68.9 ms
[16:33:59] fps: 99.3, s2st:  20.0, qt:  1.2, late:   8.1 ms
[16:33:59] fps: 71.6, s2st:  20.9, qt:  3.1, late:   0.0 ms

These four fields match the first four fields in the bulleted list below.

To capture more detailed QoS statistics, you use the launch option -tqs or -trace-qos-stats. This will capture a separate CSV log that can be opened in a spreadsheet or easily processed with a custom application, and includes the following fields:

  • Frames Per Second

  • Frame Time in CXR (ms)

  • Client Queue Time (ms)

  • Late Awaiting Latch (ms)

  • Average Video Rate (kbps)

  • Estimated Available Bandwidth (kbps)

  • Bandwidth Utilization (%)

  • Packet Loss (cumulative)

  • Jitter (us)

  • Round Trip Delay (ms)

  • Received Packets (cumulative)

  • Received Packets Dropped (cumulative)

The three fields other than fps that are output in both the Client log and the QoS CSV log are:

  • Frame Time in CXR or s2st: value is the period between when the CXR server receives the eye textures to when the eye textures leave the CloudXR library on the client, or from submit-to-submit (server to client).

  • Client Queue Time or qt: is the period that the decoded frames spend for dejittering. This is a part of the overall Frame Time in CXR value.

  • Late Awaiting Latch or late: is the period that the cxrLatchFrame() call spends waiting for frames to be available to dequeue.

Other logging feature options are available for the server and client that use launch options (see Command-Line Options for more information).

Here are some other options:

  • -d forces a PNG screenshot to be captured every 1000 frames while streaming.
    The corresponding client and server images can then be compared.
  • -lmd N specifies the number of days to retain log files, default is 5.
    As an example, -lmd 8 will delete log files that are older than 8 days when the app is launched.
  • -lmk N specifies the maximum size of the log files.
    As an example, -lmk 20480 will set the log file to truncate after surpassing 20MB.

  • -p instructs the client and server to disable privacy in streamer logging.

Crash Dump

If the server crashes, a crashdump.dmp file is created in the log directory on the server system. This file can be sent to NVIDIA for analysis.

ETL Traces

For issues that occur during streaming, you might need to collect detailed stream events in addition to the normal logs and send this information to the CloudXR support team for analysis.

Event Trace Logs (ETL) capture a trace of client, server, and network events and timings. Capturing these logs can help determine why a failure, or other issues, occurred during streaming. ETL captures are also extremely useful in debugging networking-related issues, such as low quality, stutters, and high latency. The ETL contains useful information only if the connection was successfully established and streaming started. If you cannot connect, the ETL will not help.

To enable ETL traces, add a launch option to your client or your server by completing the instructions in Command-Line Options.

Specifically, to enable ETL logging you need to add -t or -trace-stream-events to the client or server launch options.

Note

The actual logging occurs on the server, but you can request it on the client.

Warning

The trace is only completed after you exit the CloudXR server and SteamVR.

Like other server log output, the ETL trace output will be placed in the C:\Users\*user-name*\AppData\Local\NVIDIA\CloudXR\logs\ folder. In this folder, the file with .etl extension and the current date and time contains the resulting ETL trace, for example, something like 20210703-152354-NvStreamer.etl.

Warning

If you do not get an ETL trace file in the logs folder as expected, it might be related to account permissions. ETL trace capture uses a Windows system service to record events, and it requires the SteamVR process to be running in the Administrator account or from a user account that is in the Performance Log Users group.

If you are not the original Administrator account on the machine, we recommend that you have your user account added to the Performance Log Users group. The secondary account in the admin group might not work. This is the workaround for accounts that cannot capture ETL traces.

Local Event Traces (CloudXR Visual Profiler)

The local event traces can help you understand performance or stutter issues. When you specify the -tle or the -trace-local-events command-line option on the client or server, a file called Event Trace DATE TIME.json will be produced in the CloudXR log directory. The file contains timestamped events, such as when the video frame decoding started and completed on the client. There are separate files for the client and server.

Note

The trace is only a 30s capture, and it is delayed to start 10s after the client connects to ensure the stream has stabilized.

You can review the file by launching the index.html web page in TestTools\trace_vis. This step starts the CloudXR Visual Profiler in your browser, and you will be prompted to select a trace file to open. When you open this file, you should see a visualization like the following figure:

../_images/sample_trace.png

Example Traces from the CloudXR Server and Client

Example Traces from the CloudXR Server and Client shows example traces from the CloudXR server and client. Here are some of the actions you perform on the traces:

  • When you click on a colored box that represents an event, the profiler will display more detailed information.

  • When you press Control+Click multiple boxes in a row in an event, the profiler will report the total and average durations for the selected events.

  • You can also use the scroll wheel to zoom in and out of the graph and click-and-drag in the time row to shift the viewed window of data.

In your browser, you can open the HTML file in multiple tabs, or side by side in split screen, and rapidly switch between the tabs for easy comparisons.