Skip to navigation
Operations

AIStore Node Lifecycle

View as Markdown

AIStore operators can use this guide to understand node maintenance, shutdown, and decommission before changing cluster membership. These operations change node state; the diagram below shows the states and transitions.

Node lifecycle: states and transitions

To put things in perspective, this picture is about a node (not shown) in an AIStore cluster (also not shown).

Tracking it from top to bottom, first notice the state called maintenance mode. This is the gentlest way to remove a node from an operating cluster. When in maintenance, the node stops keepalive heartbeats but remains in the cluster map. A target must remain online while rebalance migrates its data. Successful completion marks it maintenance-post-rebalance in the Smap; it can then be disconnected or shut down.

Next comes shutdown. Graceful shutdown can also be achieved in a single shot, as indicated by one of the arrows on the left:

online => shutdown

When in shutdown, the node can later return and rejoin the cluster. That takes two steps, not one: restart the node first, then take it out of maintenance. In the diagram, RESTART must be understood as a deployment-specific action such as kubectl run, restarting a systemd unit, or powering the machine back on.

Normally, taking a target into maintenance or shutdown involves an intra-cluster operation called global rebalance; --no-rebalance skips it.

The third and final special state is decommission. On the normal rebalance path, the cluster attempts to migrate data from the departing target to other online targets before removing it. Rebalance errors do not always prevent removal; with --no-rebalance, migration is skipped. Decommission also entails:

  • partial or complete cleanup of the node itself; and
  • removing AIS metadata and configuration files. Local user data is removed when --rm-user-data is specified.

Needless to say, there’s no simple way back out of decommission - the proverbial point of no return. A decommissioned node does not come back by restarting it: depending on how far the cleanup progressed, it must be re-added with an explicit join or redeployed from scratch.

Table of Contents

Joining a Cluster: Discovery URL

AIStore clusters can be deployed with an arbitrary number of AIStore proxies (a.k.a. gateways). Each proxy implements RESTful APIs, both native and S3-compatible, and provides full access to user data stored in the cluster.

Each proxy collaborates with the others to perform majority-voted HA failovers; see Highly Available Control Plane. All electable proxies are functionally equivalent. The one elected as the current primary is, among other things, responsible for joining nodes to the running cluster.

To facilitate node joins in the presence of disruptive events such as:

  • network failures; and/or
  • partial or complete loss of local AIS metadata such as cluster maps,

AIStore uses the so-called original and discovery URLs in the cluster configuration. The latter is versioned, replicated, protected, and distributed solely by the elected primary.

March 2024 update: starting with v3.23, the original URL does not track the original primary. Instead, the current primary takes full responsibility for updating both URLs with a single purpose: optimizing time to join or rejoin the cluster.

When an HA event triggers automated failover, the role of primary is assumed by a different proxy, with the corresponding cluster map (Smap) update synchronized across all running nodes.

A new node, however, may still have configuration that refers to the old primary. The original and discovery URLs exist precisely to address that scenario:

$ ais config cluster proxy --json
{
    "proxy": {
        "primary_url": "https://ais-proxy-15.ais-proxy.ais.svc.cluster.local:51082",
        "original_url": "https://ais-proxy-15.ais-proxy.ais.svc.cluster.local:51082",
        "discovery_url": "https://ais-proxy.ais.svc.cluster.local:51082",
        "non_electable": false
    }
}

Cluster

There is one cluster-level lifecycle command that deserves to be called out separately:

$ ais cluster decommission --rm-user-data --yes

The above command destroys an existing cluster - completely and utterly, no questions asked. It is useful in testing, benchmarking, and other non-production environments. See --help for details.

Privileges

All lifecycle management commands and their associated APIs require administrative privileges.

Broadly, there are three ways to satisfy that requirement:

  • deploy the cluster with authentication disabled:
$ ais config cluster auth --json
{
    "auth": {
        "signature": {
            "key": "**********",
            "method": "hmac"
        },
        "required_claims": {
            "aud": null
        },
        "oidc": {
            "issuer_ca_bundle": "-",
            "allowed_iss": null
        },
        "intra_cluster": {
            "request_auth": false,
            "node_join_secret_path": "",
            "ttl": "0s",
            "nonce_window": "1m",
            "rotation_grace": "1m"
        },
        "client_auth_required": false
    }
}

Here, client_auth_required: false means protected client requests do not require authentication.

  • use the integrated AuthN server, which provides OAuth 2.0-compliant JWTs and a set of CLI auth commands to manage users, roles, and permissions; or
  • outsource authorization to a separate centralized system, often LDAP-integrated, that manages existing users, groups, and mappings.

Rebalance

Conceptually, AIStore rebalance is somewhat similar to what is often called a RAID rebuild. The underlying mechanics are different, but the high-level idea is similar: user data migrates from some nodes in a cluster to other nodes to restore the correct placement.

In AIStore, rebalancing is the system response to a lifecycle event that has already happened or is about to happen. Its singular purpose is to satisfy one governing rule:

user data must be properly located

Proper Location

For any object in a cluster, its proper location is defined by the current cluster map and, locally on each target, by the configured target mountpaths.

In that sense, the maintenance state, for instance, has its beginning when the cluster starts rebalancing, and its post-rebalancing end when the corresponding sub-state is recorded in the next Smap version and safely distributed across all nodes.

Quick Example

Given a 3-node single-gateway cluster, suppose we shut down one of the nodes:

$ ais cluster add-remove-nodes shutdown <TAB-TAB>
p[BkTqWmFd]   t[QrmZvKdN]   t[XvnGkRdM]   t[ZdpKcVnT]
$ ais cluster add-remove-nodes shutdown 't[QrmZvKdN]' -y
Started rebalance "g47" (to monitor, run 'ais show rebalance').
t[QrmZvKdN] is shutting down, please wait for cluster rebalancing to finish
Note: the node t[QrmZvKdN] is _not_ decommissioned - it remains in the cluster map and can be manually
restarted at any later time (and subsequently activated via 'stop-maintenance' operation).

Once the command is executed, notice the following:

$ ais show cluster
...
t[QrmZvKdN][x]   -   -   -   -   maintenance

At first, maintenance will show up in red, indicating a simple fact: data is expeditiously migrating from the node that is about to leave the cluster.

A visual cue that effectively says: please don’t disconnect it yet, and do not power it off.

Eventually, if you run:

$ ais show cluster --refresh 3

or simply check a few times manually, the output will report that rebalance (g47 in this example) has finished and the node t[QrmZvKdN] has gracefully left service. Simultaneously, maintenance in the show output becomes non-red:

when rebalancingafter
\{\color\{red\}maintenance\}\{\color\{cyan\}maintenance\}

On this normal shutdown path, AIS stops the target after the associated global rebalance completes successfully. If rebalance is interrupted or reports errors, the target remains in maintenance; see Incomplete Transitions.

Putting a Node in Maintenance

To temporarily take a node out of the cluster, put it in maintenance mode. Nodes in maintenance remain in the cluster map but stop participating in normal request processing.

$ ais cluster add-remove-nodes start-maintenance 't[QrmZvKdN]'
Started rebalance "g1" (to monitor, run 'ais show rebalance').
t[QrmZvKdN] is now in maintenance mode

Alternatively, you can shut the node down as part of the same workflow:

$ ais cluster add-remove-nodes shutdown 't[QrmZvKdN]'
Started rebalance "g1" (to monitor, run 'ais show rebalance').
t[QrmZvKdN] is shutting down, please wait for cluster rebalancing to finish
Note: the node t[QrmZvKdN] is _not_ decommissioned - it remains in the cluster map and can be manually
restarted at any later time (and subsequently activated via 'stop-maintenance' operation).

If the node is a target, the cluster normally rebalances after a short preparation phase. Before manually powering off a target in maintenance, see Checking Removal Status.

Batch Operations

start-maintenance, stop-maintenance, shutdown, and decommission all accept multiple nodes: NODE_ID [NODE_ID...], comma- or space-separated. TAB completion suggests the nodes not yet selected.

The batch executes as one coordinated operation. Each lifecycle phase updates the cluster map once for the entire batch. When rebalance is required, the cluster performs one RMD increment and starts one global rebalance, regardless of how many nodes you specify:

$ ais cluster add-remove-nodes start-maintenance <TAB-TAB>
p[BkTqWmFd]   t[XvnGkRdM]   t[QrmZvKdN]   t[ZdpKcVnT]   t[HbjTwLpS]   t[NwLjRbGq]
$ ais cluster add-remove-nodes start-maintenance 't[QrmZvKdN]' 't[HbjTwLpS]' --yes
Started rebalance "g1" (to monitor, run 'ais show rebalance').
t[QrmZvKdN] is now in maintenance mode
t[HbjTwLpS] is now in maintenance mode

The three remaining targets receive the migrating data - a single g1 for the pair, not one rebalance per node:

$ ais show rebalance
REB ID   NODE       TX OBJECTS   TX BYTES   RX OBJECTS   RX BYTES   START      END   STATE
g1       ZdpKcVnT   0            0B         42695        41.69MiB   12:11:28   -     Running
g1       NwLjRbGq   0            0B         42555        41.56MiB   12:11:28   -     Running
g1       XvnGkRdM   0            0B         42961        41.95MiB   12:11:28   -     Running
$ ais wait rebalance g1
Waiting for rebalance[g1] ...
Done.

Admission is all-or-nothing: if any specified node is unknown or is the current primary, the entire request is rejected and no node is touched. Conversely, once the transaction is underway, a node that disappears - keepalive-removed, for instance - does not abort it; the remaining nodes complete normally.

Operation-specific state checks also apply. start-maintenance skips a node that has already completed the same transition, and stop-maintenance skips a node that is already active. If every specified node is skipped, the command reports “nothing to do” and leaves the cluster map untouched. A node in maintenance can be advanced to shutdown or decommission; stop-maintenance refuses a node that is being decommissioned.

Advancing a node already in maintenance does not rebalance when the batch contains no active target to migrate from. A target marked maintenance-post-rebalance has already completed evacuation. If that mark is absent because rebalance was skipped, interrupted, or failed, run ais start rebalance and wait for successful completion before shutting down or removing the target. Otherwise, objects stored only on that target may become unavailable.

Incomplete Transitions

With rebalance enabled and required, each of the three removal operations follows these phases:

`start-maintenance` | `shutdown-node` | `decommission`  =>  global rebalance  =>  post-rebalance step

The first phase marks the node in the cluster map. The second attempts to migrate its data. The third records completion: for start-maintenance and shutdown, the target is marked post-rebalance; for decommission, the node is removed from the cluster map altogether.

The rebalance can be interrupted by a concurrent self-join or by another target leaving the cluster. For start-maintenance and shutdown, a rebalance that is renewed, aborted, or finishes with errors does not complete the post-rebalance transition; the target remains in maintenance. For decommission, errors can still lead to final removal, unless the rebalance was renewed or superseded. Do not treat an unsuccessful rebalance as proof that data migrated.

Skipping rebalance bypasses migration and directly finalizes the requested operation: start-maintenance leaves the target in maintenance, shutdown proceeds to shut it down, and decommission removes it from the cluster map. A target left in maintenance without a post-rebalance mark may have had rebalance skipped or interrupted; the primary cannot distinguish these cases from that state alone.

For a target in maintenance without a post-rebalance mark, the operator can:

  • repeat start-maintenance. This is accepted rather than rejected, so a retry - or a rolling-upgrade script that reissues one - does not fail. It keeps the target out of service and reapplies maintenance on the node when reachable. If no active target is specified alongside, that is all it does;
  • run stop-maintenance to clear maintenance and return the target to service, with rebalance as required;
  • advance the target to shutdown or decommission; or
  • leave it in maintenance. The next full global rebalance that completes successfully completes the transition (see below).

Such a target specified together with an active one follows the normal batch path. With automatic rebalance enabled and without --no-rebalance, that batch rebalances when active destination targets remain after the batch. Successful completion finishes the transition for both.

Specifying it together with a target that has already completed the transition changes nothing: the completed target is skipped, and the command behaves as if only the incomplete one had been specified.

Targets in maintenance without a post-rebalance mark participate in subsequent full global rebalances, sending out the data they store. A full global rebalance that completes successfully, without renewal, abort, or errors on participating targets, marks these targets post-rebalance. This applies to rebalance triggered by another membership change (e.g., a target joining the cluster) and to an explicit ais start rebalance, but not to rebalance in cleanup mode (--cleanup) or rebalance limited to a given bucket.

Targets already marked maintenance-post-rebalance have completed the transition. Subsequent rebalances neither require them to be online nor wait for their notifications.

Skipping Rebalance

Advanced usage only: --no-rebalance is not recommended for routine cluster operations. In normal operation, let AIS run rebalance automatically.

The primary recommended use case is a controlled rolling-maintenance or rolling-upgrade workflow, where nodes are taken out of service and returned in a coordinated sequence. In Kubernetes deployments, this sequencing is typically handled automatically by the AIS Kubernetes operator.

If you use --no-rebalance, the node enters maintenance immediately without waiting for data migration:

$ ais cluster add-remove-nodes start-maintenance 't[QrmZvKdN]' --no-rebalance --yes
t[QrmZvKdN] is now in maintenance mode

Keeping automatic rebalance enabled is strongly recommended, but there are cases where skipping it is safe:

  • all buckets are empty;
  • the same target is being returned with stop-maintenance after maintenance was started with --no-rebalance and no objects were added or updated during maintenance;
  • all objects can be refetched from remote backends such as remote AIS or cloud buckets, understanding that this may incur extra cloud traffic charges.

When returning multiple nodes from maintenance, name them all in a single stop-maintenance command with rebalance enabled; see Batch Operations. Batching is an alternative to skipping rebalance, not a reason to skip it.

Unchanged data during maintenance does not justify skipping migration for shutdown or decommission: removing a target without migrating its sole local copies can make those objects unavailable.

The --no-rebalance flag is available for start-maintenance, shutdown, stop-maintenance, and decommission.

One Membership Change at a Time

The primary admits one administrative membership change at a time. A second request issued while the first is still executing is refused. If the first request starts a global rebalance, the exclusion continues until that rebalance reaches a terminal state:

$ ais cluster add-remove-nodes start-maintenance 't[HbjTwLpS]' --yes
Started rebalance "g1" (to monitor, run 'ais show rebalance').
t[HbjTwLpS] is now in maintenance mode
$ ais cluster add-remove-nodes start-maintenance 't[QrmZvKdN]' --yes
Error: ErrBusy: cluster membership "start-maintenance" is currently busy (rebalance[g1] is running), please try again

The rule covers start-maintenance, stop-maintenance, shutdown, decommission, the advanced unsafe removal command, explicit join, and an operator-initiated ais start rebalance with or without --cleanup.

There are two deliberate qualifications:

  • Self-join is not serialized. A node starting or restarting and registering on its own - including normal Kubernetes restart and scale-up paths - is not subject to the administrative admission guard. It may join while a rebalance is running and cause that rebalance to be renewed.
  • An exact inverse is not exempt. Taking the same nodes back out of maintenance while their causal rebalance is running is refused like any other membership change. Wait for the rebalance to finish (ais show rebalance), then reactivate the nodes.

Do not abort a lifecycle-triggered rebalance merely to issue its inverse. Lifecycle operations are not rollback transactions: aborting rebalance does not restore the preceding Smap, can leave maintenance or shutdown transitions incomplete, and does not necessarily prevent decommission finalization.

To transition several nodes together, specify them in one command - see Batch Operations - rather than issuing requests one after another.

Clearing Maintenance State

Once a node is in maintenance mode, the cluster keeps it there until you explicitly clear that state.

If the node was shut down, restart or power it on first and wait for it to register with the primary proxy. Then run:

$ ais cluster add-remove-nodes stop-maintenance 't[QrmZvKdN]'
Started rebalance "g3" (to monitor, run 'ais show rebalance').
t[QrmZvKdN] is now active

To skip automatic rebalance, provide --no-rebalance (advanced usage only; see Skipping Rebalance).

In general, automatic rebalance should remain enabled. The same considerations listed under Skipping Rebalance apply here as well.

The node starts accepting requests again after it rejoins and the cluster clears its maintenance state. You do not have to wait for the rebalance that stop-maintenance itself starts.

Note, however, that stop-maintenance is rejected while a global rebalance is already running - including the rebalance triggered by the start-maintenance, shutdown, or decommission that put the node there. Wait for it to finish (ais show rebalance) and then reactivate.

Removing a Node from a Cluster

To permanently remove a node from the cluster, decommission it:

$ ais cluster add-remove-nodes decommission 't[QrmZvKdN]'
Started rebalance "g5" (to monitor, run 'ais show rebalance').
t[QrmZvKdN] is being decommissioned, please wait for cluster rebalancing to finish...

After the rebalance ends, the primary proxy can remove the node from the cluster map even if rebalance reported errors, unless it was renewed or superseded. On unregistering, the node cleans up its AIS metadata and removes local user data if --rm-user-data was specified. Removal from the cluster map does not establish that migration succeeded.

Skipping rebalance performs only the minimal preparation and removes the node immediately:

$ ais cluster add-remove-nodes decommission --no-rebalance 't[QrmZvKdN]'
t[QrmZvKdN] has been decommissioned (permanently removed from the cluster)

Note that decommission cleans up AIS metadata and stops the node. By contrast, shutdown only stops AIS services.

For a target, shutdown waits for the applicable rebalance to complete successfully when rebalance is required; with rebalance skipped, it proceeds without migration. For a proxy, shutdown is immediate.

Checking Removal Status

Putting a node in maintenance does not automatically power it off.

On the normal rebalance path, wait for the target’s Smap state to become maintenance-post-rebalance before powering it off. In ais show cluster target, its maintenance status changes from red to cyan when this mark is set.

To inspect the rebalance itself, replace REB_ID with the ID returned by the lifecycle command:

$ ais show rebalance REB_ID --all

Inspect that generation’s state and reported errors. An interrupted maintenance transition can also be completed by a later successful full rebalance; see Incomplete Transitions.

The REBALANCE column in ais show cluster target reflects each target’s latest rebalance snapshot, which may belong to an earlier operation. In particular, --no-rebalance starts no new generation, so an earlier finished status does not establish that the target was evacuated. Without completed evacuation, power it off only if its objects are available from other copies or remote backends.

The following output illustrates a target in maintenance after rebalance. The post-rebalance mark in the Smap confirms that its transition is complete:

$ ais show cluster target
TARGET           MEM USED %      MEM AVAIL       CAP USED %      CAP AVAIL       CPU USED %      REBALANCE    UPTIME  STATUS
QrmZvKdN         0.13%           31.28GiB        16%             2.435TiB        0.00%           finished     31m     maintenance
XvnGkRdM         0.13%           31.28GiB        16%             2.435TiB        0.12%           finished     31m     online

For decommissioning nodes, the status looks like this while rebalance is still running:

$ ais show cluster target
TARGET           MEM USED %      MEM AVAIL       CAP USED %      CAP AVAIL       CPU USED %      REBALANCE    UPTIME  STATUS
QrmZvKdN         0.13%           31.28GiB        16%             2.435TiB        0.00%           running      31m     decommission
XvnGkRdM         0.13%           31.28GiB        16%             2.435TiB        0.12%           running      31m     online

After rebalance ends, the primary proxy can remove the decommissioned node even when it reported errors (unless renewed or superseded). Removal alone does not establish successful migration:

$ ais show cluster target
TARGET           MEM USED %      MEM AVAIL       CAP USED %      CAP AVAIL       CPU USED %      REBALANCE    UPTIME
XvnGkRdM         0.13%           31.28GiB        16%             2.435TiB        0.12%           finished     31m

Summary

lifecycle operationCLIbrief description
maintenance modestart-maintenanceThe lightest way to remove a node from service. Stop keepalive heartbeats, do not insist on metadata updates, and ignore transient failures while the cluster transitions.
shutdownshutdownSame as above, plus node shutdown (aisnode exit).
decommissiondecommissionSame as above, plus partial or complete cleanup. A decommissioned node is eventually removed from the cluster map.
remove node from cluster mapais advanced remove-from-smapStrictly intended for testing and special use-at-your-own-risk scenarios. Immediately remove the node from the cluster and distribute an updated Smap with no rebalancing.
take node out of maintenancestop-maintenanceRe-enable keepalive, update the node with current cluster metadata, run global rebalance, and return the node to online.
join new nodejoinUpdate the node, synchronize current cluster metadata, and run global rebalance as needed.

All of the above are administrative membership changes. The cluster admits one at a time and refuses another while global rebalance is running; see One Membership Change at a Time. A node’s own self-join is not subject to this rule.

Assorted Notes

Normally, a starting AIS node (aisnode) uses its local configuration to contact the cluster and perform a self-join. That does not require an explicit join command or any separate administrative action.

Still, the join command is useful when the node is misconfigured. Separately, it can also be used to join a standby node - that is, a node started in standby mode; see aisnode command line.

The explicit join command is an administrative membership change and is serialized with the other operations. A node’s own self-join is not; see One Membership Change at a Time.

During rebalance, the cluster remains fully operational: users can read and write data, list, create, and destroy buckets, run jobs, and so on. In other words, none of the lifecycle operations described here requires downtime.

References