Lifecycle Restart Control

Sometimes one operator in a running pipeline needs to start over — a camera driver wedged, a model needs to rebind a device, a worker must re-run its allocation stage — without tearing down the whole application. Holoscan models this as a lifecycle restart: a bounded, permissioned transaction that rewinds one operator (and its lifecycle dependents) to a chosen forward stage and drives it back to kStart without restarting operators outside that closure. The current runtime pauses all local host dispatch at the control boundary. Operators outside the closure receive no restart callbacks. Plain downstream consumers can still receive boundary resets, as explained below.

This page covers who is allowed to trigger a restart, how an operator requests one from inside compute(), and how an external controller drives restarts through a RunSession. It builds on lifecycle callbacks (the eight stages and their ordering) and running and stopping (run_async() and RunSession). For a worked stereo graph, callback scopes and failure examples, see Lifecycle and Restart in Depth.

Complete C++ example: examples/experimental/restart_control/cpp

Deny-by-default permissions at graph scope

Nothing in a Holoscan application can restart an operator unless the graph explicitly said so. Restart authority is granted at graph scope, at authoring time, through the same graph.lifecycle() builder used for cross-operator lifecycle ordering:

graph.lifecycle()
    .allow_self_restart(worker)                     // worker may restart itself
    .allow_external_restart(worker)                 // run control outside the graph may restart worker
    .allow_restart_request(supervisor, worker);     // supervisor may request worker's restart

The three grants are independent and narrow:

Two prerequisites are frozen into the plan at compile time, and both fail closed at run time if missing:

1. The grants above. An unauthorized request returns kLifecycleRestartUnauthorized; it does not degrade to a weaker action.
2. The compile-time opt-in. The plan must be compiled with the lifecycle_transactions option set in CompileOptions (see compile and check). The two failure modes differ: when the option is absent, compilation succeeds and every dedicated restart API path returns kLifecycleTransactionsDisabled at run time; when the option is present, all nine of its capacity fields must be positive and at least the compiler-derived minima, or compilation itself is rejected with LIFECYCLE_TRANSACTION_CAPACITY_* diagnostics.

Dedicated lifecycle restart is also profile-gated at compile admission: only the standard, robotics, and research profiles admit it, and enabling the opt-in on a plan containing a real_time or regulated region fails compilation rather than degrading at run time. Parameter-origin restart inherits the narrower active-mutation profile gate, so the admitted sets differ:

The parameter field still uses the same transaction kernel after admission; the narrower row is the parameter-mutation profile policy, not a different restart algorithm. See runtime parameter updates for the complete parameter gate.

This is the same fail-closed philosophy as the rest of the runtime: the plan records exactly what is allowed, and anything not recorded is rejected as a typed error value rather than silently permitted.

Requesting a restart from compute()

An operator with allow_self_restart requests its own restart through the ExecutionContext it receives in compute(). The request names a forward stage to rewind to — kConfigure through kStart; the teardown stages kStop and kRelease are never valid entry points:

holoscan::expected<void, holoscan::Error> compute(holoscan::ExecutionContext& ctx) override {
  if (device_needs_rebind_) {
    HOLOSCAN_TRY(auto receipt,
                 ctx.request_restart(holoscan::LifecycleRestartOptions{
                     .from_stage = holoscan::LifecycleStage::kAllocate}));
    (void)receipt;   // LifecycleRestartReceipt: run-scoped identity, not a waiter
  }
  return {};
}

A two-argument overload, ctx.request_restart(target, options), requests the restart of an authorized other operator (requires allow_restart_request(requester, target) on the graph). In both forms:

• The call is nonblocking and returns expected<LifecycleRestartReceipt, Error>: acceptance only queues the request and installs an activation fence — a restart is never applied mid-compute. A self-restart begins after the current compute() callback retires. For a cross-target request, the current implementation also waits for every local owner to reach the boundary with no activation in progress. The requester must finish its callback too. An idle target does not restart concurrently with that requester’s compute().
• The receipt is a small run-scoped identity you can hand to external observation code. It records the accepted request and its requested stage; it does not wait, cancel, or grant any extra authority by itself.
• LifecycleRestartOptions::reason is a registered audit code (LifecycleRestartReason), not free text — restart requests carry no hot-path strings.
• Rejections come back as typed errors — kLifecycleRestartUnauthorized (including a self-restart the plan did not admit), kLifecycleInvalidRestartStage, kLifecycleRestartQueueFull, and friends — without changing any state.

External control: LifecycleControlHandle

Code outside the run — an application supervisor, a health monitor, an operations console — drives restarts through a capability minted from the RunSession. You first bind plan operator identities to the current run with session.operator_ref(...), then mint a handle with an explicit access policy naming the targets and stages this handle may touch:

#include <holoscan/core/lifecycle_control.hpp>
holoscan::RunSession session = holoscan::run_async(plan);
// Bind the plan-level reference to this run generation.
const std::array<holoscan::OperatorRef, 1> targets{ *session.operator_ref(plan_ref) };
const std::array<holoscan::LifecycleStage, 1> stages{ holoscan::LifecycleStage::kStart };
const holoscan::LifecycleAccessPolicy policy{ .allowed_targets = targets,
                                              .allowed_from_stages = stages };
auto handle = session.lifecycle_control(policy);   // expected<LifecycleControlHandle, Error>
if (!handle) { /* kLifecycleTransactionsDisabled, kLifecycleRestartUnauthorized, ... */ return; }
auto receipt = handle->request_restart(
    targets[0],
    holoscan::LifecycleRestartOptions{ .from_stage = holoscan::LifecycleStage::kStart });
if (!receipt) { /* typed rejection; no state changed */ return; }
// Observe the transaction: nonblocking inspect(), or the one blocking op, await().
const auto view = handle->await(*receipt, std::chrono::seconds{5});
// On success, inspect view->state: terminal outcomes include kPublished,
// kCanceled, kRolledBack, and kFailedStopped.

The handle exposes four operations:

Restriction-only, by design

A LifecycleControlHandle is a restriction-only capability. The policy you mint it with — and any policy passed to restrict() — can only narrow the authority the plan admitted at compile time; it can never add a target or a stage. Empty allowed_targets or allowed_from_stages spans deny everything. This means you can hand a tightly scoped handle to a subsystem (say, “may restart the camera operator from kStart, nothing else”) without that subsystem being able to escalate.

The handle is an opaque, process-local value: copying it preserves exactly the same authority, it retains no RunSession, plan, or operator, and accepted receipts remain inspectable until the last handle copy is destroyed — even after the originating session has closed. Never serialize, transport, or persist a handle; if you need a durable record, keep the receipt or a diagnostic record instead.

What a restart actually does

A restart is localized, but it is not always one operator. Three semantics determine its shape — all decided by the compiler and executor, never by the caller:

Closure scope. The compiler follows lifecycle edges forward from the target across all stages, including Stop/Release ordering and rate-derived dependencies. Facts (produces/requires_dependency) and graph.lifecycle().before(...) both contribute. The resulting callback closure cannot be narrowed by a request. An incoming edge from an outside provider does not pull that provider in. A plain data edge does not add callback members, although a direct consumer outside the closure can receive boundary handling.

Widening. Resolve the closure first, then the entry stage. A forward-stage dependency whose two endpoints are in the closure can widen a later request to its earlier stage. The receipt retains the requested stage. LifecycleRestartReceiptView from inspect()/await() reports the effective stage and transaction lineage. Each plan closure’s stage_resolutions explains the compiler’s decision. Compatible queued requests may coalesce over the union of their closures at the earliest resolved entry stage.

Re-entry depth. The effective stage, after widening and coalescing, determines teardown and the forward suffix:

A Start restart does not invoke Release, but that does not guarantee resource retention: a Stop callback can release resources itself. Allocate and earlier entries include Release and replay resource acquisition. Stop and Release are never valid requested entry stages (kLifecycleInvalidRestartStage).

Fencing and receiver boundaries. Acceptance installs an activation fence. In-flight activations retire before every local owner reaches the control boundary. The fence cancels pending candidates and retires frozen sources for callback members and their direct same-process consumers. Callback members then run teardown and re-entry. On successful publication, both sets reset their selectors/readiness and advance their receiver Tag generations. The remaining operators retain theirs. Only callback members receive lifecycle callbacks. All local dispatch pauses during the transaction. This is not a distributed barrier or recursive traversal of downstream data edges.

A callback closure can happen to cover the whole graph. There is no separate all-operators restart selector or interprocess atomic restart guarantee. To reconstruct all operator instances, stop and create a new RunSession. See running and stopping.

One restart transaction, end to end

A dedicated request returns a receipt. A parameter-origin update returns its own update identity. Acceptance means queued work, not completed reconfiguration. Off-callback control code observes a dedicated receipt with inspect()/await(). A request submitted from compute() can hand its receipt to that code, but the callback cannot wait through the control handle itself.

The restart outcome map shows where the transaction can resume or fail the run. Preparation checks drain/completion state, conditionally cancels and joins the shared channel service, fences candidates, and prepares tensor/rate state before any lifecycle callback.

RolledBack does not restore canceled candidates, queued data or arbitrary device side effects. See the worked recovery example for the callback ledger and the distinction between cleanup and restoration.

Stop closes ingress and cancels queued dedicated work. A dedicated transaction already executing retains its actual outcome before terminal stop. Parameter-origin work additionally checks stop before candidate publication. An await() timeout (kTimeout) only ends that caller’s wait. It does not cancel or mutate the transaction. Later observation can report the result.

Restarts triggered by parameter updates

A restart does not always begin with an explicit request. A typed-parameter field declared .lifecycle_mutable(stage) submits into the same transaction kernel when patched at run time: the accepted batch reserves a candidate parameter generation and requests re-entry at the earliest invalidated stage across the batch’s lifecycle-mutable fields. The candidate uses kStart as its lifecycle barrier, but it becomes publicly visible only after the whole coalesced transaction publishes. The full treatment — admission, guardrails, result states, and correlation IDs — is on runtime parameter updates.

Parameter views inside restart callbacks

Lifecycle callbacks see a phase-selected parameter view during a restart. The lifecycle callbacks page introduces the mechanism; this is the exact selection:

Recap

• Restart authority is deny-by-default: grant it at graph scope with allow_self_restart, allow_external_restart, or allow_restart_request(requester, target), and compile with lifecycle_transactions enabled.
• Operators request restarts from compute() via ctx.request_restart(LifecycleRestartOptions{.from_stage = ...}); only forward stages (kConfigure–kStart) are valid entry points, and lifecycle callbacks can never submit one.
• External controllers call session.lifecycle_control(policy) to mint a LifecycleControlHandle with request_restart / inspect / await / restrict — a restriction-only capability that can narrow but never broaden the plan’s admitted authority.
• A restart executes over the target’s compiler-frozen lifecycle-dependency closure with the exact teardown/forward matrix above: kStop always, plus kRelease when the compiler-resolved effective stage is kAllocate or earlier. The request may widen toward kConfigure, and execution is fenced to the control safe point — never applied mid-compute(); operators outside the closure receive no lifecycle callbacks, while all local dispatch pauses during the transaction.
• Requests are nonblocking and return expected<LifecycleRestartReceipt, Error>; the receipt carries the requested stage, while inspect()/await() return the LifecycleRestartReceiptView with the effective stage and lineage. Read failures as typed error values.
• A run-time patch to a .lifecycle_mutable(stage) parameter field enters the same transaction kernel: the candidate generation publishes only after the whole transaction reaches kPublished.

Next: change an operator’s typed parameters in a running session in Runtime parameter updates.