Lifecycle and Restart in Depth

A sensor pipeline needs coordination at several levels. Cameras must finish initialization before fusion can compute. Fusion needs suitable capture timestamps. A failed device may need to restart while other operators retain their state. Each requirement has its own mechanism.

This walkthrough follows two cameras, a fusion operator and an unrelated monitor. It derives their behavior from lifecycle declarations, then connects that model to the shipped camera + IMU example. The stereo diagrams are illustrative predictions, not a recorded application trace. For runnable callback and restart examples, see examples/lifecycle_callbacks/cpp and examples/experimental/restart_control.

Declarations, a compiled plan, and a running session

Constructors, setup() and contract() describe an operator. Authoring and compilation can replay them, so keep resource acquisition and externally visible work in lifecycle callbacks. Compilation validates the declarations and freezes stage checkpoints, same-stage ordering graphs, restart authority and closures. No lifecycle callback runs during compilation.

A new RunSession reconstructs runtime operator instances and bindings, performs startup, then admits computation. A targeted restart reuses those instances. Stopping and creating another session reconstructs fresh instances instead.

A successful or rolled-back restart returns to computation within the same session. A session can also stop without restarting. Terminal stop tears down and joins the session.

Why a fixed set of stages?

A stage is a shared ordering and cleanup checkpoint. The runtime knows which stages precede computation, which forward suffix a restart must repeat, and which completed work is eligible for cleanup. Application-specific device state belongs inside the operator and its hooks.

These are useful responsibilities, not an assertion that every shipped operator uses every stage. A simple operator may do all its setup and cleanup in start()/stop(). Advanced hooks have the signature LifecycleStatus method(LifecycleContext&) noexcept.

An operator with no hooks writes no lifecycle code. Its checkpoints still take part in stage ordering and complete as skipped. You may omit an optional hook. If you register it, its failure still affects execution. kRetryableFailure classifies a failure but does not schedule an automatic retry. See Lifecycle callbacks for registration, context services and status values.

Two cameras: separate data edges from lifecycle edges

Left and Right send frames to Fusion. Monitor has no data connection. Fixed nominal rates keep rate-derived restart dependencies out of this first example.

Solid arrows carry payloads. Dashed arrows order lifecycle checkpoints at one named stage. Each stage has its own graph. Neither kind transfers resource ownership or synchronizes camera exposure. The Stop view is a deliberate variation, separate from the baseline example.

Now declare that each camera produces an armed fact, and Fusion requires both facts at kArm. The following declaration fragments omit ports, class definitions and callback bodies. LeftCamera, RightCamera and Fusion are illustrative application classes.

// Shared dependency keys, declared at namespace scope.
inline constexpr holoscan::LifecycleDependencyKey kLeftArmed{"left.armed"};
inline constexpr holoscan::LifecycleDependencyKey kRightArmed{"right.armed"};
// In LeftCamera::setup(): RightCamera produces kRightArmed instead.
spec.lifecycle()
    .stage(holoscan::LifecycleStage::kAllocate, &LeftCamera::allocate)
    .stage(holoscan::LifecycleStage::kArm, &LeftCamera::arm)
    .stage(holoscan::LifecycleStage::kStop, &LeftCamera::stop_camera)
    .stage(holoscan::LifecycleStage::kRelease, &LeftCamera::release)
    .produces(holoscan::LifecycleStage::kArm, kLeftArmed);
// In Fusion::setup():
spec.lifecycle()
    .stage(holoscan::LifecycleStage::kArm, &Fusion::arm)
    .requires_dependency(holoscan::LifecycleStage::kArm, kLeftArmed)
    .requires_dependency(holoscan::LifecycleStage::kArm, kRightArmed);

Monitor registers no hooks. A dependency fact is a compiled ordering declaration, not a runtime message or a promise to poll until a string key appears. Missing or duplicate producers and same-stage cycles are rejected during compilation.

The global barriers already order different stages, so declared dependencies connect checkpoints within one stage.

Startup is stage-major

Every operator completes Configure before any enters Discover, and so on through Start. Within a stage, callbacks execute serially in stable topological order, on their assigned execution lanes.

The filled marks are the five startup hooks declared above. An empty mark is a successful skipped checkpoint. At Arm, both camera callbacks precede Fusion’s callback. Monitor has no dependency edge and skips its checkpoint.

Can Left compute while Fusion is still in Arm? No. Initial startup completes the last Start checkpoint and readiness/source initialization before opening compute dispatch and restart ingress. Once running, ordinary activations follow their own contracts.

Three meanings of “in sync”

For stereo data, OnAligned with Exact member selectors matches FIFO heads within an inclusive capture-time spread. For camera + IMU, OnWindow with BracketedWindow selects a complete neighborhood around consecutive camera anchors. Neither follows automatically from an Arm dependency. See Temporal contracts and Advanced temporal patterns.

Derive restart membership before its entry stage

The compiler follows lifecycle edges forward, across every stage, to freeze each target’s callback closure. This includes Stop/Release ordering edges and any rate-derived dependencies. The closure is independent of the requested entry stage. Data-flow edges alone do not establish callback membership.

The second panel previews the deliberate Stop ordering described below.

With only the Arm ordering, a Left restart has C = {Left, Fusion}. Right’s incoming edge to Fusion does not run backwards. Monitor is outside C, and this baseline has no direct data consumer outside C.

Next, the compiler compares the requested stage with forward-stage dependencies whose two endpoints are inside C. An earlier internal edge widens the entry stage. An incoming dependency from an outside provider does not alone widen it.

The receipt keeps the requested stage. inspect()/await() report the effective stage. The plan’s resolution records explain the widening reason.

Couple cameras only for a real ordering requirement

Suppose Fusion must stop using both cameras’ resources before either camera stops. Add Fusion → Left and Fusion → Right at Stop. Those are separate from the camera → Fusion edges at Arm.

Each stage graph is acyclic. Across stages, forward reachability now connects all three operators, so restarting any one includes Left, Right and Fusion. A Start request widens to Arm. This is a consequence of real lifetime ordering, not a separate restart-group API. Declare dependencies for actual lifetime ordering.

Who pauses and who receives callbacks?

Return to the baseline graph with only Arm dependencies. Add a Detector that consumes Fusion’s output through an ordinary same-process data edge, with no lifecycle dependency. Give Detector hooks of its own so that “no callbacks during a Left restart” is meaningful.

Detector’s data edge adds boundary handling without adding lifecycle callbacks.

The runtime waits until every local owner reaches a control boundary with no activation in progress. It then prepares the transaction, runs callbacks for C, and resumes owners only after a resumable outcome. An unrelated operator is not reconstructed, but uninterrupted progress is not promised. This is a local session boundary, not distributed quiescence of interprocess neighbors.

Preparation completes before C’s lifecycle callbacks:

A failure here produces kFailedStopped and fails the run before any callback. The channel service is shared across the session. Even a B receiver near end-of-stream can fail the fence.

Which stages repeat?

The executor uses the effective entry stage. Compatible queued requests can coalesce into a union closure with the earliest shared entry stage. Several receipts can therefore identify the same transaction.

A Start-entry transaction omits Release. That does not guarantee that Stop preserves application resources: an operator may release resources in Stop and recreate them in Start. Declare important teardown ordering at Stop/Release explicitly. Ordinary teardown follows those stage graphs. Rollback unwinds completed candidate work in reverse completion order. Independent Stop-peer ordering in initial/final session transitions need not be the restart graph’s ordering.

Choose the control operation

Dedicated restart needs compile-time transaction capacity and explicit self, external or requester-to-target grants. A restriction-only control handle cannot add authority absent from the plan. Lifecycle-mutable parameter fields use parameter-origin authority after admission. They do not require the dedicated restart grant. Parameter control requires update/scope authority and its own supported profile. Keep the restart and parameter profile matrices separate.

compute() can submit a nonblocking request and hand its receipt to off-callback control code. Only control code outside runtime callbacks uses a LifecycleControlHandle to inspect or await it. LifecycleContext cannot submit or await restarts. Acceptance does not mean completion. A caller-side wait timeout does not cancel the transaction.

See Restart control for the complete APIs and Runtime parameter updates for update-sequence/result observation. Parameter updates do not return dedicated restart receipts. Atomic parameter batches are same-operator only: restarting both cameras does not atomically change both cameras’ parameter values.

A model change has resource ownership

Consider an application-defined Detector with a scalar uint32_t model_id that selects entries 0 and 1 in a fixed compatible catalog. Declare the field with .range(0U, 1U) and .lifecycle_mutable(holoscan::LifecycleStage::kAllocate). Allocate owns model resources, WarmUp prepares them, Stop quiesces active use, and Release destroys the owned resource set. This is a design example, not a claim that the stock inference operator implements this contract.

For Detector’s own update, C is {Detector} if it has no outgoing lifecycle or rate dependencies. With no downstream consumer, B is empty. The incoming Fusion data edge does not pull the cameras or Fusion into the closure. Detector was in B for a Left restart. Membership depends on the target.

An out-of-range model ID is rejected with ParamUpdateStatus::kOutOfBounds before lifecycle submission. To reason about restoration, instead use a valid catalog ID whose Allocate succeeds but WarmUp fails:

1. Pre-restart Stop and Release use the old published value.
2. Candidate Allocate succeeds, but candidate WarmUp fails.
3. Candidate cleanup invokes Release for the completed allocation. The failed WarmUp callback must clean its own unrecorded partial acquisition with RAII.
4. RestoreOld reruns Allocate → WarmUp → Arm → Start using the old value.
5. If cleanup, restoration and runtime recovery succeed, the update rolls back and the run continues with the old published configuration.

Restoration loads the old resource again because teardown already released it. There is no uninterrupted fallback instance or automatic reversal of external device effects. Stock InferenceOp does not expose a lifecycle-mutable model-catalog selector. Its Stop/Start restart reinitializes the same configuration. A different model map or port contract needs supported reconfiguration or a new plan/session.

Name the counter before asking whether it resets

External endpoints retain their own publication-sequence authority. These counters describe different state and should be observed separately.

Pre-restart teardown sees old published parameters. Restart-forward sees the candidate for a patched owner and retained current values for dependency-only members. Cleanup sees the view that created the work. Restoration sees the old view. Final teardown sees the then-current published view. Readiness, rate and resource generations are further identities, not aliases for one universal epoch.

Failure is not always recoverable

A stop sign marks a step whose failure ends the run. Restoration re-executes the old suffix after completed candidate work has been cleaned up. It does not undo arbitrary device effects. Only participating parameter candidates change values on publication.

RolledBack does not restore canceled candidates or queue contents. Its warning carries detail.transaction_id. FailedStopped ends the run without relaunching owners. A parameter candidate is retired without publication on that path. Partial runtime publication may already have changed internal state, so a failed run does not guarantee intact generations.

A stop request cancels queued work and closes ingress. An already executing dedicated restart retains its actual outcome before terminal stop. Parameter-origin work has an additional stop-before-publication fence.

Inspect the plan, then observe the right result

The public plan does not expose every runtime receiver-binding record. For a plain downstream consumer, derive the data relationship from the graph and observe Tags and callback absence. See Plan tooling for inspection.

For a restart that seems broader than expected, check forward lifecycle reachability before the requested stage. Lifecycle-selected nominal rates can add closure dependencies. A fixed-rate example does not exercise that path. If resources need rebuilding earlier, declare the actual Allocate ordering or lifecycle-mutable stage. Camera mode changes are device-specific: the owning operator must quiesce acquisition, invalidate the right resources and restart with valid timestamps. Restart does not promise physical exposure alignment. Keep such reconfiguration outside a CUDA capture region, which rejects restart targets or closure members.

Check your prediction

Try answering these questions from the graph:

For a camera/IMU pipeline, predict the interval samples, missing policy and end-of-stream behavior using the worked bracket.

Recap

• Fixed stages provide shared ordering, restart suffixes and cleanup rules. Sparse hooks let each operator implement only its relevant work.
• Startup is global and stage-major. Restart callback scope is compiled from forward dependencies, then its effective entry stage is resolved.
• Local dispatch pauses at the control boundary. Callback members, direct boundary consumers and remaining operators have different restart effects.
• Publication, restoration and failed-stop outcomes preserve different state. Name each parameter, Tag and output counter explicitly.
• Lifecycle readiness, hardware synchronization and temporal data selection solve different problems. Declare each requirement through its own mechanism.