Advanced Features

This chapter describes the more advanced features of Fleet Command including administrative settings, security overrides, and remote management.

Fleet Command Settings

This section helps to understand Fleet Command settings. To view the settings, navigate to the Settings page in Fleet Command.

../_images/managing-your-setup-14.png
  • System: Get the latest Fleet Command system image and source code for the system image.

  • Remote Management: Enable or disable Remote Console and system reboot option and modify the remote console timeouts. Enable or disable remote application access and modify the remote application access timeouts. For more information on Remote Console or remote application access, refer to Remote Management.

  • Deployment Security Overrides: Enable or disable selecting security overrides when creating deployments. For more information, refer to Creating a Deployment.

    Important

    Enabling or disabling Deployment Security Overrides in Settings only affects new deployments created after the setting is applied. Disabling security overrides will not affect existing deployments that have overrides configured. These deployments will need to be deleted and recreated with security overrides disabled.

    ../_images/managing-your-setup-15.png

    If Fleet Command was previously enabled for Device Access, you would now need to enable Deployment Security Overrides to achieve similar functionality when creating new deployments.

    ../_images/managing-your-setup-16.png
  • Logs: Enable All Logs to capture all system and location logs and enable application and deployment logging. For more information on logs, refer to Fleet Command Logs.

Remote Management

Fleet Command provides the following ways to remotely access and manage your systems and applications deployed at the edge site.

  • Remote Console : to create remote shell sessions

  • Remote Application Access : to access web-based services

Remote Console

Fleet Command allows you to access your system remotely with the help of Remote Console.

Remote Console allows Fleet Command administrators to remotely start and access shell sessions for each system. One console can be created for each system, up to 20 consoles per organization. Multiple administrators can access the system through the remote console. Remote Console will create a shell session within the system for each user.

Note

  • Remote Console is available in Fleet Command Stack 1.3.0 and above.

  • To access the console remotely, you will need TLS 443 outbound connection to be allowed from the system. Refer to Edge Site Requirements for additional information.

Enabling Remote Console

  1. Within the Fleet Command User Interface, navigate to Settings.

    ../_images/troubleshooting-fleet-command-01.png
  2. Toggle the slider under Remote Management to enable Remote Console.

    ../_images/troubleshooting-fleet-command-02.png
  3. If necessary, configure the timeouts as required for your use case.

    • Max is the maximum time allowed for remote console sessions; after max time is reached all the active remote console sessions will be closed automatically.

    • Inactivity is the timeout for idle shell sessions, each user will get their own shell session, and inactivity time will be tracked independently for each user.

    Note

    Session time should range from 2 minutes up to an hour. It is recommended that idle time should match or be less than the session time.

Accessing Remote Console

  1. Access to the Remote Console is available from the system ellipsis.

    ../_images/troubleshooting-fleet-command-03.png
  2. When you click on Start Remote Console, it will ask you to authenticate again for additional security. Login again to access Remote Console.

    ../_images/troubleshooting-fleet-command-28.png
  3. After successful authentication, a session is created in the background. A pop-up will display the remote console with the system name.

    ../_images/troubleshooting-fleet-command-04.png
  4. Click Open to launch the Remote Console.

    ../_images/troubleshooting-fleet-command-45.png

    Note

    Depending on the Fleet Command version, you will be presented with the Remote Console interface or prompted to update the location version.

    ../_images/troubleshooting-fleet-command-46.png
  5. When the Remote Console launches, it will redirect to a new browser session or tab, as shown below.

    ../_images/troubleshooting-fleet-command-29.png
  6. To switch to the root user for the troubleshooting, follow the steps as shown below and enter the Fleet Command Administrator password of your edge system for the password prompts.

    1
    2
    3
    4
    5
    rcuser@ip-172-31-41-77:~$ su admin
    Password:
    admin@ip-172-31-41-77:/home/rcuser$ sudo su
    [sudo] password for admin:
    root@ip-172-31-41-77:/home/rcuser$
    

    Note

    • The rcuser and admin user are the basic users that can run nvidia-smi and basic Linux commands without sudo.

    • The root user can run all the commands like kubectl to debug the edge system.

  7. Right-click on the shell to display the available options for Remote Console.

    ../_images/troubleshooting-fleet-command-31.png
  8. Select the text that you would like to copy and paste using the remote console menu for troubleshooting as shown below.

    ../_images/troubleshooting-fleet-command-32.png ../_images/troubleshooting-fleet-command-33.png
  9. Another option is you can use the Paste from the browser from the remote console menu to paste into a shell.

    ../_images/troubleshooting-fleet-command-34.png ../_images/troubleshooting-fleet-command-35.png
  10. If you don’t have a physical keyboard, you can use the Onscreen Keyboard option with Remote Console 2.0.

    ../_images/troubleshooting-fleet-command-36.png ../_images/troubleshooting-fleet-command-37.png ../_images/troubleshooting-fleet-command-38.png
  11. For container runtime troubleshooting, you can use CTR CLI or Container Runtime Interface (CRI) CLI.

    ../_images/troubleshooting-fleet-command-39.png ../_images/troubleshooting-fleet-command-40.png

    Note

    • Use Kubernetes Cheat Sheet for any deployment issues from Remote Console.

    • The command history of the Remote Console will be purged when the console shell session is closed or max session timeout.

  12. If the remote session is closed for any reason, you can Open the remote console session from the Remote Consoles banner under each location. You can click End the active remote consoles to close all active remote console sessions for this system.

    ../_images/troubleshooting-fleet-command-08.png
  13. For each Fleet Command organization, a maximum of 20 active remote console sessions are allowed. If more than 20 remote console sessions are attempted, you will receive the following error message.

    ../_images/troubleshooting-fleet-command-10.png

Rebooting a System

Important

While a system is rebooting, applications will stop running, and applications will start after the system rebooted.

  1. To reboot a system, click on the ellipsis under a System and select Reboot System.

    ../_images/troubleshooting-fleet-command-11.png ../_images/troubleshooting-fleet-command-12.png
  2. You might see the Message below when the system is rebooting.

    ../_images/troubleshooting-fleet-command-13.png
    • If the remote console session is active, you will be prompted with the below pop-up.

      ../_images/troubleshooting-fleet-command-41.png
  3. The UI will show the results of the reboot, followed by a success message.

    ../_images/troubleshooting-fleet-command-14.png

Disabling Remote Console

Important

Disabling Remote Console within Fleet Command Settings will immediately close all remote consoles within the organization.

  1. Within the Fleet Command User Interface, navigate to Settings.

    ../_images/troubleshooting-fleet-command-42.png
  2. Toggle the slider under Remote Management.

    ../_images/troubleshooting-fleet-command-43.png

If there are any active remote consoles, you will be prompted before disabling the remote management.

../_images/troubleshooting-fleet-command-44.png

Remote Application Access

Fleet Command allows you to access web-based application services running on edge systems remotely from your local machine. Remote application access is available via a unique URL to users with Fleet Command administrator and operator roles. Remote application access features the following:

  • A configurable time allowance to access services. When the time expires, remote access to services for that application will automatically end. This greatly simplifies resource management and frees up available remote sessions for other services.

  • Accessible by multiple users in multiple locations. Regardless of the remote access origin, all admin and operator users in the same organization have access to the remote services.

Configuring your Remote Application

To support remote application access, you must explicitly configure applications to allow remote access via the appProtocol field in the Kubernetes service.

Fleet Command invokes your application using a mapping to the root location of your web service. If your application requires additional paths for access, Fleet Command recommends that you configure a redirect from the application root location to the full path of your application.

The following example shows a mapping of the full path of a web application to the resulting remote application service URL :

Sample remote application URL mapping
  • The web application root location http://<ip_address_of_node>:31115 is mapped to https://<fc_remaac_location>-31115.rmsession.<org>.egx.nvidia.com where <fc_remaac_location> is the name of your system and <org> is your organization name.

  • Fleet Command will invoke your application using the mapped root location - https://<fc_remaac_location>-31115.rmsession.nvidia.egx.nvidia.com, as in this example.

  • If you have already set up a redirect from http://<ip_address_of_node>:31115 to http://<ip_address_of_node>:31115/WebRTCApp/play.html?name=videoanalytics, your application will launch automatically. Fleet Command recommends using NGINX Ingress to set up the redirect and to access the application via the Ingress NodePort.

  • If the redirect is not in place, you will need to manually append the rest of the application path to the mapped root location in the browser.

Refer to Fleet Command Helm Chart Requirements for more information on configuring the application.

Enabling Remote Application Access

Fleet Command administrators can enable remote application access in Fleet Command > Settings. Under Remote Management, toggle the Enable remote application access setting.

Enabling remote application access

Configuring Access Timeout

The remote application access timeout is the duration where application services are accessible. When remote application access is enabled in Settings, administrators can adjust the maximum timeout. The default timeout is 3 hours. To adjust the timeout, navigate to Fleet Command > Settings. Under Remote Management, set the maximum hours and minutes in the Remote application access timeout field.

Remote application access timeout

Starting Remote Application Access

This section is for Fleet Command users with administrator or operator roles and describes the steps for starting remote application access.

You must choose a location first to view a list of available application services.

  1. Go to Fleet Command > Deployments in the navigation menu.

    In the deployments page, choose a deployment and click on Expand arrow arrow to view the list of locations.

    View locations in deployments
  2. To view the details for a location, click on “…” (ellipsis) menu and select View Details.

    View details for a location
  3. The Details pane displays the location information and application services deployed at this location. The list displays all services for this application, but only web-based services with unique URLs are remotely accessible.

    Application service names are generated using the following pattern:

    default/<application name>/<protocol>
    

    Web-based services are identified by the ‘http’ protocol.

    Deployment details pane

    Important

    If the application service is not explicitly enabled for external access, and/or the appProtocol field is not applied to the application service, the service will still be listed with the ‘http’ protocol by default. However, note that only actual web-based services will function properly. Accessing the link for non-web-based services will fail.

    If an application does not have services, you will see the message No application services found in the deployment Details pane for that location.

    Remote application access no services found

    To start remote application access for a service, click on a Service Name link.

    The Remote Application Access confirmation dialog appears with the allotted time for accessing the remote service. When the time expires, users are unable to access the services, and all running sessions will end.

    Click Continue to open the application service or Cancel to exit.

    Start remote application access confirmation dialog

    While the connection is in progress, you will see the following progress dialog:

    Start remote application access progress dialog

    After the progress dialog dismisses, you will see a Remote Application Access banner in the deployment section. The banner shows currently opened services and their expiration time in hours and minutes. A badge appears on the top right corner of the page showing the number of currently opened services.

    You can click on Open to start or view the service or End to stop the remote application access.

    Remote application access banner and badge
  4. The application service will open in a new browser tab. The following image shows an example remote service:

    Accessing a sample remote application

    Note

    In some cases, the remote application may not open automatically in a browser tab. If the window does not open automatically, check the pop-up blocker for your web browser, and configure it to allow pop-ups from Fleet Command. Click the Open button to open the window manually.

    Important

    If your application does not load correctly, make sure you follow the Configuring your Remote Application instructions.

Maximum services

You may enable remote access up to twenty services within an org. If the maximum is reached, you will see a Remote Application Access Limit Reached dialog when starting a remote service in the deployments Details pane. You must close a service that’s not in use before you can open another.

Remote application access limit reached

Ending Remote Application Access

Fleet Command users with administrator or operator roles can end access to services for all users in the org.

To end access to remote application services, follow these steps:

  1. Click Fleet Command > Deployments in the navigation menu.

  2. In the Remote Application Access banner, click on End to stop the service.

    Remote application access banner
  3. A confirmation dialog appears warning the action will end the connection for other users. Click End to proceed or Cancel to exit.

    Ending remote application access

    If there are no remaining open access sessions for this deployment, the banner will dismiss on its own.

Disabling Remote Application Access

Administrators can disable remote application access for all users in the org.

To disable remote access, navigate to Fleet Command > Settings. Under Remote Management, toggle off the Enable remote application access setting.

Disabling remote application access

If remote services are in use, a confirmation dialog appears. Click Disable to end access for all users or Cancel to exit.

Disable remote application access confirmation dialog

Once remote access is disabled, users are unable to start any remote access for any deployments. The Service Name links in the Details pane are not clickable.

Remote application service links unclickable

Security Overrides

If the option to allow Security Overrides in Fleet Command Settings is enabled, additional configuration options are available that may turn off enforcement of specific security policies and enable the application to access more system hardware and software. These security overrides will reduce system security and should be used at the Administrator’s own risk. NVIDIA recommends that a minimum of security overrides be applied only for testing and troubleshooting to ensure maximum security of the system.

The following security overrides are available:

  • Enable all overrides

  • Allow system device access: This allows access to devices mounted at /dev.

  • Allow HostPaths: This enables access to any hostpath. If this option is disabled (by default), only /mnt, /opt, /tmp, and /etc/localtime are allowed.

  • Allow HostPath mount via PersistentVolumes: Enable the application to use hostPath for Persistent Volumes.

  • Allow HostNamespace (HostIPC and HostPID): Enable application access to the host system processes and inter-process interactions.

  • Allow Linux capabilities: Enable applications to use any Linux capabilities.

  • Allow PrivilegedContainers: This option allows privileged containers that run as root to run.

There may be certain combinations of locations and security overrides that are not supported. For details, refer to the following:

  • During Deployment creation, if another location is added after security overrides other than the Allow system device access option are applied, and the second location that was added is on a version older than 1.9.3, you will be prompted to disable all other security overrides except Allow system device access as per below.

../_images/managing-your-setup-17.png
  • During Deployment creation, when you choose two locations with different versions, by default, all other security overrides will be disabled except Allow system device access.

../_images/managing-your-setup-18.png
  • All security overrides will be grayed out and uneditable when editing the deployment. Security overrides cannot be changed after the Deployment has been created. To change security overrides, the deployment must be removed and recreated.

Note

Security override options selected when a Deployment was first created cannot be changed when editing a deployment irrespective of the Deployment Security Overrides option in Fleet Command Settings.

../_images/managing-your-setup-19.png
  • When editing the deployment, if the current deployment with location(s) is enabled with security overrides other than Allow system device access, all locations on versions older than 1.9.3 will be unselectable.

../_images/managing-your-setup-20.png

Important

If Security Overrides are applied, a message will be displayed in the Location(s) and Deployments.

../_images/managing-your-setup-21.png ../_images/managing-your-setup-22.png

Application Configuration

  1. Using the standard template format (same as Helm Charts), users can override existing configuration from the values YAML file or set a value. For example, setting the cameras to be used:

    ../_images/deploying-to-the-edge-03.png
  2. To deploy the application to the systems at a specific location, check the box at the bottom and click Deploy.

    Note

    If multiple deployments/applications are assigned to a single location, you will need to confirm that they do not conflict. Some examples to keep in mind are service/application ports, resources (over subscribing), duplicate applications, or other application conflicts within the same namespace. Ensure you search for the application Helm chart and not the container, model, or collections.

Note

To perform deployment updates or removals, refer to the Managing Deployments section.

Multi-Instance GPU Configuration

Important

If NVIDIA Multi-Instance GPU (MIG) is enabled on the system(s) with an active deployment, the currently running application(s) will be stopped during MIG configuration. If the Application Compatibility mode is On, the running application(s) will restart after MIG is enabled. If Application Compatibility mode is Off, the application(s) may not restart successfully and require reconfiguration and redeployment. More information on Application Compatibility mode can be found later on, and information about configuring applications for MIG can be found in the Application Guide.

NVIDIA Multi-Instance GPU (MIG) allows supported GPUs to be partitioned and used across multiple applications. Fleet Command allows you to enable and configure MIG on supported systems directly from the UI on location versions 1.5.2 or later.

Note

MIG capability is only supported on NVIDIA A30 and A100 GPUs. For unsupported GPUs, the System Details panel will display that no MIG-capable GPUs are present.

../_images/mig-01.png

For additional information on MIG, refer to the Multi-Instance GPU User Guide.

To view additional information about MIG, click System Details using the ellipsis.

../_images/mig-02.png

System Details will appear on the right side of the window displaying the system information, MIG configuration, advanced networking details.

../_images/mig-03.png

Fleet Command allows you to configure specific MIG profiles for A30 and A100. The supported MIG profiles are listed in the following table.

MIG Available Options

Supported GPUs

2 MIGs of 3c.20gb

A100

3 MIGs of 2c.10gb

A100

7 MIGs of 1c.5gb

A100

2 MIGs of 2c.12gb

A30

4 MIGs of 1c.6gb

A30

To configure MIG, from the system details page under the MIG section, select the GPU, choose the MIG profile from the drop-down menu, select the checkbox to understand the application disruption, and click on Save to apply the changes. Once you apply the MIG configuration, it will take a minute or two to reflect the changes on the system.

../_images/mig-04.png

Accessing Your System

Different configuration combinations for systems with more than one MIG-capable GPU may impact application compatibility mode.

  • If MIG is not enabled on all GPUs in a multi-GPU system, Application Compatibility mode will be Off, and a specific application configuration will be required to use MIG.

  • If MIG is enabled on all GPUs, but with different MIG profile sizes, the Application Compatibility mode will also be Off.

  • If MIG is enabled on all GPUs with the same MIG profile sizes, Application Compatibility mode will be On.

Refer to the Application Guide for additional information on Application Compatibility mode and configuring applications for MIG.

../_images/mig-05.png ../_images/mig-06.png