Upgrade NetQ Agent Using LCM

The lifecycle management (LCM) feature enables you to upgrade to NetQ 3.3.0 on switches with an existing NetQ Agent 2.4.x-3.2.1 release using the NetQ UI. You can upgrade only the NetQ Agent or upgrade both the NetQ Agent and the NetQ CLI at the same time. Up to five jobs can be run simultaneously; however, a given switch can only be contained in one running job at a time.

The upgrade workflow includes the following steps:

Upgrades can be performed from NetQ Agents of 2.4.x and 3.0.x-3.2.x releases. Lifecycle management does not support upgrades from NetQ 2.3.1 or earlier releases; you must perform a new installation in these cases. Refer to Install NetQ Agents.

Prepare for a NetQ Agent Upgrade

Prepare for NetQ Agent upgrade on switches as follows:

  1. Click (Upgrade) in the workbench header.

  2. Add the upgrade images.

  3. Optionally, specify a default upgrade version.

  4. Verify or add switch access credentials.

  5. Optionally, create a new switch configuration profile.

Your LCM dashboard should look similar to this after you have completed the above steps:

  1. Verify or add switch access credentials.

  2. Configure switch roles to determine the order in which the switches get upgraded.

  3. Upload the Cumulus Linux install images.

Perform a NetQ Agent Upgrade

You can upgrade NetQ Agents on switches as follows:

  1. In the Switch Management tab, click Manage on the Switches card.

  2. Select the individual switches (or click to select all switches) with older NetQ releases that you want to upgrade. Filter by role (on left) to narrow the listing and sort by column heading (such as hostname or IP address) to order the list in a way that helps you find the switches you want to upgrade.

  3. Click (Upgrade NetQ) above the table.

    From this point forward, the software walks you through the upgrade process, beginning with a review of the switches that you selected for upgrade.

  1. Verify that the number of switches selected for upgrade matches your expectation.

  2. Enter a name for the upgrade job. The name can contain a maximum of 22 characters (including spaces).

  3. Review each switch:

    • Is the NetQ Agent version between 2.4.0 and 3.2.1? If not, this switch can only be upgraded through the switch discovery process.
    • Is the configuration profile the one you want to apply? If not, click Change config, then select an alternate profile to apply to all selected switches.

You can apply different profiles to switches in a single upgrade job by selecting a subset of switches (click checkbox for each switch) and then choosing a different profile. You can also change the profile on a per switch basis by clicking the current profile link and selecting an alternate one.

Scroll down to view all selected switches or use Search to find a particular switch of interest.

  1. After you are satisfied with the included switches, click Next.

  2. Review the summary indicating the number of switches and the configuration profile to be used. If either is incorrect, click Back and review your selections.

  1. Select the version of NetQ Agent for upgrade. If you have designated a default version, keep the Default selection. Otherwise, select an alternate version by clicking Custom and selecting it from the list.

By default, the NetQ Agent and CLI are upgraded on the selected switches. If you do not want to upgrade the NetQ CLI, click Advanced and change the selection to No.

  1. Click Next.

  2. Several checks are performed to eliminate preventable problems during the upgrade process.

These checks verify the following when applicable:
  • Selected switches are not currently scheduled for, or in the middle of, a Cumulus Linux or NetQ Agent upgrade
  • Selected version of NetQ Agent is a valid upgrade path
  • All mandatory parameters have valid values, including MLAG configurations
  • All switches are reachable
  • The order to upgrade the switches, based on roles and configurations

If any of the pre-checks fail, review the error messages and take appropriate action.

If all of the pre-checks pass, click Upgrade to initiate the upgrade job.

  1. Watch the progress of the upgrade job.
You can watch the detailed progress for a given switch by clicking .
  1. Click to return to Switches listing.

    For the switches you upgraded, you can verify the version is correctly listed in the NetQ_Version column. Click to return to the lifecycle management dashboard.

    The NetQ Install and Upgrade History card is now visible in the Job History tab and shows the status of this upgrade job.

To upgrade the NetQ Agent on one or more switches, run:

netq-image name <text-job-name> [netq-version <text-netq-version>] [upgrade-cli True | upgrade-cli False] hostnames <text-switch-hostnames> [config_profile <text-config-profile>]

This example creates a NetQ Agent upgrade job called upgrade-cl430-nq330. It upgrades the spine01 and spine02 switches with NetQ Agents version 3.3.0.

cumulus@switch:~$ netq lcm upgrade name upgrade-cl430-nq330 netq-version 3.3.0 hostnames spine01,spine02

Analyze the NetQ Agent Upgrade Results

After starting the upgrade you can monitor the progress in the NetQ UI. Progress can be monitored from the preview page or the Upgrade History page.

From the preview page, a green circle with rotating arrows is shown on each switch as it is working. Alternately, you can close the detail of the job and see a summary of all current and past upgrade jobs on the NetQ Install and Upgrade History page. The job started most recently is shown at the top, and the data is refreshed periodically.

If you are disconnected while the job is in progress, it may appear as if nothing is happening. Try closing (click ) and reopening your view (click ), or refreshing the page.

Monitor the NetQ Agent Upgrade Job

Several viewing options are available for monitoring the upgrade job.

  • Monitor the job with full details open:

  • Monitor the job with only summary information in the NetQ Install and Upgrade History page. Open this view by clicking in the full details view; useful when you have multiple jobs running simultaneously.

    When multiple jobs are running, scroll down or use the filters above the jobs to find the jobs of interest:

    • Time Range: Enter a range of time in which the upgrade job was created, then click Done.
    • All switches: Search for or select individual switches from the list, then click Done.
    • All switch types: Search for or select individual switch series, then click Done.
    • All users: Search for or select individual users who created an upgrade job, then click Done.
    • All filters: Display all filters at once to apply multiple filters at once. Additional filter options are included here. Click Done when satisfied with your filter criteria.

    By default, filters show all of that items of the given filter type until it is restricted by these settings.

  • Monitor the job through the NetQ Install and Upgrade History card in the Job History tab. Click twice to return to the LCM dashboard.

Sample Successful NetQ Agent Upgrade

This example shows that all four of the selected switches were upgraded successfully. You can see the results in the Switches list as well.

Sample Failed NetQ Agent Upgrade

This example shows that an error has occurred trying to upgrade two of the four switches in a job. The error indicates that the access permissions for the switches are invalid. In this case, you need to modify the switch access credentials and then create a new upgrade job.

If you were watching this job from the LCM dashboard view, click View on the NetQ Install and Upgrade History card to return to the detailed view to resolve any issues that occurred.

To view the progress of upgrade jobs, run:

netq lcm show upgrade-jobs netq-image [json]
netq lcm show status <text-lcm-job-id> [json]

You can view the progress of one upgrade job at a time. To do so, you first need the job identifier and then you can view the status of that job.

This example shows all upgrade jobs that are currently running or have completed, and then shows the status of the job with a job identifier of job_netq_install_7152a03a8c63c906631c3fb340d8f51e70c3ab508d69f3fdf5032eebad118cc7.

cumulus@switch:~$ netq lcm show upgrade-jobs netq-image json
[
    {
        "jobId": "job_netq_install_7152a03a8c63c906631c3fb340d8f51e70c3ab508d69f3fdf5032eebad118cc7",
        "name": "Leaf01-02 to NetQ330",
        "netqVersion": "3.3.0",
        "overallStatus": "FAILED",
        "pre-checkStatus": "COMPLETED",
        "warnings": [],
        "errors": [],
        "startTime": 1611863290557.0
    }
]

cumulus@switch:~$ netq lcm show status netq-image job_netq_install_7152a03a8c63c906631c3fb340d8f51e70c3ab508d69f3fdf5032eebad118cc7
NetQ Upgrade FAILED

Upgrade Summary
---------------
Start Time: 2021-01-28 19:48:10.557000
End Time: 2021-01-28 19:48:17.972000
Upgrade CLI: True
NetQ Version: 3.3.0
Pre Check Status COMPLETED
Precheck Task switch_precheck COMPLETED
	Warnings: []
	Errors: []
Precheck Task version_precheck COMPLETED
	Warnings: []
	Errors: []
Precheck Task config_precheck COMPLETED
	Warnings: []
	Errors: []


Hostname          CL Version  NetQ Version  Prev NetQ Ver Config Profile               Status           Warnings         Errors       Start Time
                                            sion
----------------- ----------- ------------- ------------- ---------------------------- ---------------- ---------------- ------------ --------------------------
leaf01            4.2.1       3.3.0         3.2.1         ['NetQ default config']      FAILED           []               ["Unreachabl Thu Jan 28 19:48:10 2021
                                                                                                                         e at Invalid
                                                                                                                         /incorrect u
                                                                                                                         sername/pass
                                                                                                                         word. Skippi
                                                                                                                         ng remaining
                                                                                                                         10 retries t
                                                                                                                         o prevent ac
                                                                                                                         count lockou
                                                                                                                         t: Warning:
                                                                                                                         Permanently
                                                                                                                         added '192.1
                                                                                                                         68.200.11' (
                                                                                                                         ECDSA) to th
                                                                                                                         e list of kn
                                                                                                                         own hosts.\r
                                                                                                                         \nPermission
                                                                                                                         denied,
                                                                                                                         please try a
                                                                                                                         gain."]
leaf02            4.2.1       3.3.0         3.2.1         ['NetQ default config']      FAILED           []               ["Unreachabl Thu Jan 28 19:48:10 2021
                                                                                                                         e at Invalid
                                                                                                                         /incorrect u
                                                                                                                         sername/pass
                                                                                                                         word. Skippi
                                                                                                                         ng remaining
                                                                                                                         10 retries t
                                                                                                                         o prevent ac
                                                                                                                         count lockou
                                                                                                                         t: Warning:
                                                                                                                         Permanently
                                                                                                                         added '192.1
                                                                                                                         68.200.12' (
                                                                                                                         ECDSA) to th
                                                                                                                         e list of kn
                                                                                                                         own hosts.\r
                                                                                                                         \nPermission
                                                                                                                         denied,
                                                                                                                         please try a
                                                                                                                         gain."]

Reasons for NetQ Agent Upgrade Failure

Upgrades can fail at any of the stages of the process, including when backing up data, upgrading the NetQ software, and restoring the data. Failures can also occur when attempting to connect to a switch or perform a particular task on the switch.

Some of the common reasons for upgrade failures and the errors they present:

ReasonError Message
Switch is not reachable via SSHData could not be sent to remote host “192.168.0.15.” Make sure this host can be reached over ssh: ssh: connect to host 192.168.0.15 port 22: No route to host
Switch is reachable, but user-provided credentials are invalidInvalid/incorrect username/password. Skipping remaining 2 retries to prevent account lockout: Warning: Permanently added ‘<hostname-ipaddr>’ to the list of known hosts. Permission denied, please try again.
Switch is reachable, but a valid Cumulus Linux license is not installed1587866683.880463 2020-04-26 02:04:43 license.c:336 CRIT No license file. No license installed!
Upgrade task could not be runFailure message depends on the why the task could not be run. For example: /etc/network/interfaces: No such file or directory
Upgrade task failedFailed at- <task that failed>. For example: Failed at- MLAG check for the peerLink interface status
Retry failed after five attemptsFAILED In all retries to process the LCM Job