NICo supports two tenant-facing repair paths and provider-facing repair workflows. Start here to choose the right workflow.
Online repair is the least disruptive path. It is requested through the Machine update API, keeps the tenant assignment in place, and uses a repair health override to keep the instance in Repairing.
Full repair is the disruptive path. It is requested through the Instance delete/release API with a machine health issue. The tenant gives up the instance, and NICo prevents the machine from returning to normal allocation until repair and validation are complete.
An instance cannot be released for full repair while it is still in online repair. If online repair is active and the issue now requires full repair, clear online repair first, wait for the instance to return to Ready, and then release the instance for full repair.
When clearing online repair because the repair failed, update the instance labels before releasing it for full repair. A label such as onlineRepair.status: Failed keeps the failure visible to tenant and operator tooling while the instance is being escalated.
Repair system integration is the provider-side behavior behind full repair. It explains how tenant-reported-issue, repair-request, repair tenants, and manual provider actions fit together.
The repair tenant workflow is the operator runbook for the middle of the full repair process: targeted instance creation into a dedicated repair tenant, machine repair, repair outcome labeling, and repair tenant release.
In a multi-Tenant Site, NICo allows certain Tenants to be designated as privileged Tenants who can carry out repair workflows. A privileged Tenant can access the following REST API endpoints that a regular Tenant cannot:
Tenant privileges are described in terms of Capabilities. The capabilities of a Tenant can be viewed by calling the GET /v2/org/{org}/nico/tenant/current REST API endpoint and inspecting the capabilities field. To elevate a Tenants privileges so they can access the above endpoints, the Tenant must have the targetedInstanceCreation capability enabled.
In service account mode, the targetedInstanceCreation capability is granted to Service Account Tenant when GET /v2/org/{org}/nico/service-account/current REST API endpoint is called.
At present turning this capability on for regular Tenants (who are not part of a Service Account org) is not supported via the REST API. However the feature is in active development, relevant issue can be tracked here.
NOTE: Privileged Tenants still need Network Allocations from Provider in order to create Instances.
Reporting an issue with the underlying Machine when deleting the Instance doesn’t require any special privileges. All Tenants can perform this action.
Enabling and disabling online repair for an Instance does not require any special privileges. All Tenants can perform this action.
Use the individual runbooks for exact payloads and validation rules.
Tenant admins should read:
Provider operators and repair automation owners should also read: