> This page is for version 5.0 (Latest) (default).
> For other versions, use one of these documentation indexes:
> - 5.0 (Latest) (default): https://docs.nvidia.com/holoscan/sdk-user-guide/5-0-latest/llms.txt
> - ltsb-2.0: https://docs.nvidia.com/holoscan/sdk-user-guide/ltsb-2-0/llms.txt
> - 4.6: https://docs.nvidia.com/holoscan/sdk-user-guide/4-6/llms.txt
> - 4.5: https://docs.nvidia.com/holoscan/sdk-user-guide/4-5/llms.txt
> - 4.4: https://docs.nvidia.com/holoscan/sdk-user-guide/4-4/llms.txt
> - 4.3: https://docs.nvidia.com/holoscan/sdk-user-guide/4-3/llms.txt
> - Archives: https://docs.nvidia.com/holoscan/sdk-user-guide/archives/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/holoscan/sdk-user-guide/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/holoscan/sdk-user-guide/_mcp/server.

A sensor pipeline needs coordination at several levels. Cameras must finish
initialization before fusion can compute. Fusion needs suitable capture
timestamps. A failed device may need to restart while other operators retain
their state. Each requirement has its own mechanism.

This walkthrough follows two cameras, a fusion operator and an unrelated
monitor. It derives their behavior from lifecycle declarations, then connects
that model to the shipped
[camera + IMU example](/holoscan/sdk-user-guide/examples/camera-imu-control).
The stereo diagrams are illustrative predictions, not a recorded application
trace. For runnable callback and restart examples, see
[`examples/lifecycle_callbacks/cpp`](https://github.com/nvidia-holoscan/holoscan-sdk/tree/v5.0.0/examples/lifecycle_callbacks/cpp)
and [`examples/experimental/restart_control`](https://github.com/nvidia-holoscan/holoscan-sdk/tree/v5.0.0/examples/experimental/restart_control).

> **You will learn**
>
> * Why fixed stages and optional hooks work together
> * Why initial startup waits for every operator, while restart callbacks have a smaller scope
> * How lifecycle edges determine restart membership and the effective entry stage
> * What pauses, what gets reset, and what a successful or failed restart preserves
> * How lifecycle readiness differs from sensor synchronization and temporal selection

> **Warning**
>
> **Beta.** Lifecycle restart and active parameter-update APIs may change in
> upcoming releases. Receiver-boundary handling and temporal timing details
> called out below describe the current implementation. They do not establish
> scheduling latency or hardware synchronization guarantees.

## Declarations, a compiled plan, and a running session

Constructors, `setup()` and `contract()` describe an operator. Authoring and
compilation can replay them, so keep resource acquisition and externally visible
work in lifecycle callbacks. Compilation validates the declarations and freezes
stage checkpoints, same-stage ordering graphs, restart authority and closures.
No lifecycle callback runs during compilation.

A new `RunSession` reconstructs runtime operator instances and bindings, performs
startup, then admits computation. A targeted restart reuses those instances.
Stopping and creating another session reconstructs fresh instances instead.

<title id="lid-session-title">
  What happens before, during and after one RunSession
</title>

<desc id="lid-session-desc">
  Author and Compile happen before any session. Replayable declarations become a frozen plan, and no lifecycle callback runs yet. The plan starts a RunSession, drawn as a band. Inside the band, the session reconstructs operators and bindings, runs startup through every stage until ready, and then computes. A restart runs at a control boundary and reuses the same instances. The restart returns to compute on Published or RolledBack. FailedStopped ends the run. Stop tears down and joins the session. Optionally, creating another RunSession reconstructs fresh instances from the same plan.
</desc>

<defs>
  <path d="M 0,0 L 8,4 L 0,8 Z" fill="currentColor" />
</defs>

<rect x="14" y="0" width="140" height="52" rx="8" />

<rect x="186" y="0" width="140" height="52" rx="8" />

<g fill="currentColor">
  <circle cx="40" cy="146" r="5.5" />

  <circle cx="40" cy="186" r="5.5" />

  <circle cx="40" cy="226" r="5.5" />

  <circle cx="40" cy="402" r="5.5" />
</g>

Author

<text x="26" y="41">
  declarations
</text>

Compile

<text x="198" y="41">
  frozen plan
</text>

<text x="14" y="76">
  no lifecycle callbacks yet
</text>

<text x="264" y="80">
  plan
</text>

RunSession

<text x="56" y="151">
  Reconstruct operators and bindings
</text>

<text x="56" y="191">
  Startup: every stage, then ready
</text>

Compute

Restart: same instances

<text x="70" y="302">
  runs at a control boundary
</text>

<text x="70" y="324">
  returns on Published or RolledBack
</text>

<text x="94" y="352">
  FailedStopped ends the run
</text>

<text x="56" y="407">
  Stop: teardown and join
</text>

<text x="52" y="449">
  create another session
</text>

Next RunSession

<text x="28" y="504">
  fresh instances from the same plan
</text>

A successful or rolled-back restart returns to computation within the same
session. A session can also stop without restarting. Terminal stop tears down
and joins the session.

## Why a fixed set of stages?

A stage is a shared ordering and cleanup checkpoint. The runtime knows which
stages precede computation, which forward suffix a restart must repeat, and
which completed work is eligible for cleanup. Application-specific device state
belongs inside the operator and its hooks.

| Stage        | Typical application work                                   |
| ------------ | ---------------------------------------------------------- |
| `kConfigure` | Validate selected configuration and derive cold intent     |
| `kDiscover`  | Query capabilities, modes or dimensions                    |
| `kAllocate`  | Acquire the resources owned by this lifecycle pass         |
| `kWarmUp`    | Prepare those resources for steady execution               |
| `kArm`       | Bind generation-scoped readiness and external capabilities |
| `kStart`     | Enable active behavior                                     |
| `kStop`      | Quiesce active use                                         |
| `kRelease`   | Release owned resources                                    |

These are useful responsibilities, not an assertion that every shipped operator
uses every stage. A simple operator may do all its setup and cleanup in
`start()`/`stop()`. Advanced hooks have the signature
`LifecycleStatus method(LifecycleContext&) noexcept`.

An operator with no hooks writes no lifecycle code. Its checkpoints still take
part in stage ordering and complete as skipped. You may omit an optional hook.
If you register it, its failure still affects execution.
`kRetryableFailure` classifies a failure but does not schedule an automatic retry. See
[Lifecycle callbacks](/holoscan/sdk-user-guide/building-operators/lifecycle-callbacks)
for registration, context services and status values.

## Two cameras: separate data edges from lifecycle edges

Left and Right send frames to Fusion. Monitor has no data connection. Fixed
nominal rates keep rate-derived restart dependencies out of this first example.

<title id="lid-views-title">
  Data, Arm ordering and Stop ordering are separate graphs
</title>

<desc id="lid-views-desc">
  Three views of the same four operators in fixed positions: Left, Fusion and Right in a row, and Monitor below Right. Data: Left and Right each send payloads to Fusion. Monitor has no data connection. Arm ordering: Left and Right each precede Fusion at Arm. Left, Right and Fusion register Arm hooks. Monitor has no Arm hook, so its Arm checkpoint is skipped. Stop ordering, a coupled variation used only if Fusion must stop first: Fusion precedes Left and Right at Stop.
</desc>

<defs>
  <path d="M 0,0 L 8,4 L 0,8 Z" fill="currentColor" />
</defs>

<rect x="10" y="28" width="64" height="44" rx="8" />

<rect x="134" y="28" width="72" height="44" rx="8" />

<rect x="266" y="28" width="64" height="44" rx="8" />

<rect x="242" y="88" width="88" height="34" rx="8" />

<rect x="10" y="170" width="64" height="44" rx="8" />

<rect x="134" y="170" width="72" height="44" rx="8" />

<rect x="266" y="170" width="64" height="44" rx="8" />

<rect x="242" y="230" width="88" height="34" rx="8" />

<rect x="10" y="332" width="64" height="44" rx="8" />

<rect x="134" y="332" width="72" height="44" rx="8" />

<rect x="266" y="332" width="64" height="44" rx="8" />

<rect x="242" y="392" width="88" height="34" rx="8" />

<circle cx="236" cy="153" r="5.5" />

<circle cx="42" cy="170" r="5.5" />

<circle cx="170" cy="170" r="5.5" />

<circle cx="298" cy="170" r="5.5" />

Data

Arm ordering

<text x="246" y="158">
  Arm hook
</text>

Stop ordering: coupled variation

<text x="14" y="320">
  only if Fusion must stop first
</text>

no Arm hook: skipped

<text x="42" y="55">
  Left
</text>

<text x="170" y="55">
  Fusion
</text>

<text x="298" y="55">
  Right
</text>

<text x="286" y="110">
  Monitor
</text>

<text x="42" y="197">
  Left
</text>

<text x="170" y="197">
  Fusion
</text>

<text x="298" y="197">
  Right
</text>

<text x="286" y="252">
  Monitor
</text>

<text x="104" y="185">
  Arm
</text>

<text x="236" y="185">
  Arm
</text>

<text x="42" y="359">
  Left
</text>

<text x="170" y="359">
  Fusion
</text>

<text x="298" y="359">
  Right
</text>

<text x="286" y="414">
  Monitor
</text>

<text x="104" y="347">
  Stop
</text>

<text x="236" y="347">
  Stop
</text>

Solid arrows carry payloads. Dashed arrows order lifecycle checkpoints at one
named stage. Each stage has its own graph. Neither kind transfers resource
ownership or synchronizes camera exposure. The Stop view is a deliberate
variation, separate from the baseline example.

Now declare that each camera produces an armed fact, and Fusion requires both
facts at `kArm`. The following declaration fragments omit ports, class definitions
and callback bodies. `LeftCamera`, `RightCamera` and `Fusion` are illustrative
application classes.

```cpp
// Shared dependency keys, declared at namespace scope.
inline constexpr holoscan::LifecycleDependencyKey kLeftArmed{"left.armed"};
inline constexpr holoscan::LifecycleDependencyKey kRightArmed{"right.armed"};

// In LeftCamera::setup(): RightCamera produces kRightArmed instead.
spec.lifecycle()
    .stage(holoscan::LifecycleStage::kAllocate, &LeftCamera::allocate)
    .stage(holoscan::LifecycleStage::kArm, &LeftCamera::arm)
    .stage(holoscan::LifecycleStage::kStop, &LeftCamera::stop_camera)
    .stage(holoscan::LifecycleStage::kRelease, &LeftCamera::release)
    .produces(holoscan::LifecycleStage::kArm, kLeftArmed);

// In Fusion::setup():
spec.lifecycle()
    .stage(holoscan::LifecycleStage::kArm, &Fusion::arm)
    .requires_dependency(holoscan::LifecycleStage::kArm, kLeftArmed)
    .requires_dependency(holoscan::LifecycleStage::kArm, kRightArmed);
```

Monitor registers no hooks. A dependency fact is a compiled ordering declaration,
not a runtime message or a promise to poll until a string key appears. Missing
or duplicate producers and same-stage cycles are rejected during compilation.

The global barriers already order different stages, so declared dependencies
connect checkpoints within one stage.

## Startup is stage-major

Every operator completes Configure before any enters Discover, and so on through
Start. Within a stage, callbacks execute serially in stable topological order,
on their assigned execution lanes.

<title id="lid-startup-title">
  Startup runs stage by stage across every operator
</title>

<desc id="lid-startup-desc">
  A grid of six startup stages, from Configure down to Start, for four operators: Left, Right, Fusion and Monitor. Every operator finishes a stage before any operator enters the next one. Callbacks run only for registered hooks: Left and Right at Allocate, and Left, Right and Fusion at Arm. Every other checkpoint, including all of Monitor's, completes as skipped without calling operator code. Arm has two levels. Left and Right are in the first level, together with Monitor's skipped checkpoint, and Fusion follows both cameras in the second level. Callbacks run one at a time, level by level. After Start and readiness setup finish, compute and restart ingress open.
</desc>

<defs>
  <path d="M 0,0 L 8,4 L 0,8 Z" fill="currentColor" />
</defs>

<text x="122" y="16">
  Left
</text>

<text x="180" y="16">
  Right
</text>

<text x="238" y="16">
  Fusion
</text>

<text x="296" y="16">
  Monitor
</text>

<text x="14" y="49">
  Configure
</text>

<text x="14" y="81">
  Discover
</text>

<text x="14" y="113">
  Allocate
</text>

<text x="14" y="145">
  WarmUp
</text>

<text x="14" y="179">
  Arm
</text>

<text x="14" y="243">
  Start
</text>

<circle cx="122" cy="44" r="6" />

<circle cx="180" cy="44" r="6" />

<circle cx="238" cy="44" r="6" />

<circle cx="296" cy="44" r="6" />

<circle cx="122" cy="76" r="6" />

<circle cx="180" cy="76" r="6" />

<circle cx="238" cy="76" r="6" />

<circle cx="296" cy="76" r="6" />

<circle cx="238" cy="108" r="6" />

<circle cx="296" cy="108" r="6" />

<circle cx="122" cy="140" r="6" />

<circle cx="180" cy="140" r="6" />

<circle cx="238" cy="140" r="6" />

<circle cx="296" cy="140" r="6" />

<circle cx="296" cy="174" r="6" />

<circle cx="122" cy="238" r="6" />

<circle cx="180" cy="238" r="6" />

<circle cx="238" cy="238" r="6" />

<circle cx="296" cy="238" r="6" />

<circle cx="112" cy="331" r="6" />

<g fill="currentColor">
  <circle cx="122" cy="108" r="7" />

  <circle cx="180" cy="108" r="7" />

  <circle cx="122" cy="174" r="7" />

  <circle cx="180" cy="174" r="7" />

  <circle cx="238" cy="204" r="7" />

  <circle cx="20" cy="331" r="6" />
</g>

<text x="14" y="292">
  After Start and readiness setup:
</text>

<text x="14" y="312">
  compute and restart ingress open
</text>

<text x="32" y="336">
  callback
</text>

<text x="124" y="336">
  skipped checkpoint
</text>

<text x="14" y="358">
  One callback at a time, level by level
</text>

The filled marks are the five startup hooks declared above. An empty mark is a
successful skipped checkpoint. At Arm, both camera callbacks precede Fusion's
callback. Monitor has no dependency edge and skips its checkpoint.

**Can Left compute while Fusion is still in Arm? No.** Initial startup completes
the last Start checkpoint and readiness/source initialization before opening
compute dispatch and restart ingress. Once running, ordinary activations follow
their own contracts.

### Three meanings of “in sync”

| Requirement                                             | Mechanism                                                          | What it does not establish                      |
| ------------------------------------------------------- | ------------------------------------------------------------------ | ----------------------------------------------- |
| All operators ready before computation                  | Lifecycle stage barriers and ordering facts                        | Simultaneous physical sensor capture            |
| Cameras expose together or share a calibrated time base | Device triggering, synchronization and timestamp/clock integration | Which queued samples fusion selects             |
| Fusion receives a suitable pair or interval             | Temporal trigger, selectors and missing policy                     | Hardware synchronization or equal capture rates |

For stereo data, `OnAligned` with `Exact` member selectors matches FIFO heads
within an inclusive capture-time spread. For camera + IMU, `OnWindow` with
`BracketedWindow` selects a complete neighborhood around consecutive camera
anchors. Neither follows automatically from an Arm dependency. See
[Temporal contracts](/holoscan/sdk-user-guide/building-operators/temporal-contracts)
and [Advanced temporal patterns](/holoscan/sdk-user-guide/advanced/advanced-temporal-patterns).

## Derive restart membership before its entry stage

The compiler follows lifecycle edges **forward**, across every stage, to freeze
each target's callback closure. This includes Stop/Release ordering edges and
any rate-derived dependencies. The closure is independent of the requested entry
stage. Data-flow edges alone do not establish callback membership.

The second panel previews the deliberate Stop ordering described below.

<title id="lid-reach-title">
  Restart membership follows lifecycle edges forward from Left
</title>

<desc id="lid-reach-desc">
  Two panels with the same operator positions. Left, Fusion and Right sit in a row, and Monitor sits below Right with no edges. Baseline: Left and Right each have an Arm edge into Fusion. Following lifecycle edges forward from Left reaches Fusion only. Right's edge points into Fusion and is not followed backward. C is Left and Fusion. Right and Monitor are outside C. Coupled variation: Fusion must stop before both cameras, so Stop edges run from Fusion to Left and from Fusion to Right. Forward from Left, the Arm edge reaches Fusion and Fusion's Stop edge reaches Right. C is Left, Fusion and Right. Monitor is outside C.
</desc>

<defs>
  <path d="M 0,0 L 8,4 L 0,8 Z" fill="currentColor" />
</defs>

followed forward from Left

<rect x="10" y="60" width="64" height="52" rx="8" />

<rect x="134" y="60" width="72" height="52" rx="8" />

<rect x="10" y="266" width="64" height="52" rx="8" />

<rect x="134" y="266" width="72" height="52" rx="8" />

<rect x="266" y="266" width="64" height="52" rx="8" />

<rect x="266" y="60" width="64" height="52" rx="8" />

<rect x="242" y="156" width="88" height="34" rx="8" />

<rect x="242" y="362" width="88" height="34" rx="8" />

Restart Left: baseline

<text x="14" y="214">
  C: Left, Fusion. Outside C: Right, Monitor
</text>

Restart Left: Fusion stops first

<text x="14" y="420">
  C: Left, Fusion, Right. Outside C: Monitor
</text>

<text x="42" y="91">
  Left
</text>

<text x="170" y="91">
  Fusion
</text>

<text x="298" y="91">
  Right
</text>

<text x="286" y="178">
  Monitor
</text>

<text x="104" y="73">
  Arm
</text>

<text x="236" y="73">
  Arm
</text>

<text x="42" y="297">
  Left
</text>

<text x="170" y="297">
  Fusion
</text>

<text x="298" y="297">
  Right
</text>

<text x="286" y="384">
  Monitor
</text>

<text x="104" y="279">
  Arm
</text>

<text x="236" y="279">
  Arm
</text>

<text x="104" y="316">
  Stop
</text>

<text x="236" y="316">
  Stop
</text>

With only the Arm ordering, a Left restart has C = `{Left, Fusion}`. Right's incoming edge to Fusion does
not run backwards. Monitor is outside C, and this baseline has no direct data
consumer outside C.

Next, the compiler compares the requested stage with forward-stage dependencies
whose **two endpoints are inside C**. An earlier internal edge widens the entry
stage. An incoming dependency from an outside provider does not alone widen it.

| Target / requested stage | Callback closure | Effective entry |
| ------------------------ | ---------------- | --------------- |
| Left / Start             | Left, Fusion     | Arm             |
| Fusion / Start           | Fusion           | Start           |
| Left / Discover          | Left, Fusion     | Discover        |

The receipt keeps the requested stage. `inspect()`/`await()` report the effective
stage. The plan's resolution records explain the widening reason.

### Couple cameras only for a real ordering requirement

Suppose Fusion must stop using both cameras' resources before either camera
stops. Add Fusion → Left and Fusion → Right **at Stop**. Those are separate
from the camera → Fusion edges at Arm.

Each stage graph is acyclic. Across stages, forward reachability now connects all
three operators, so restarting any one includes Left, Right and Fusion. A Start
request widens to Arm. This is a consequence of real lifetime ordering, not a
separate restart-group API. Declare dependencies for actual lifetime ordering.

## Who pauses and who receives callbacks?

Return to the baseline graph with only Arm dependencies. Add a Detector that
consumes Fusion's output through an ordinary same-process data edge, with no
lifecycle dependency. Give Detector hooks of its own so that “no callbacks
during a Left restart” is meaningful.

<title id="lid-scope-title">
  One Left restart affects three disjoint sets
</title>

<desc id="lid-scope-desc">
  Left, Fusion and Right sit in a row. Detector sits below Fusion and Monitor below Right. Left and Right each send data to Fusion and have an Arm edge into Fusion. Fusion sends data down to Detector, with no lifecycle edge. Restarting Left follows the Arm edge forward to Fusion, so C is Left and Fusion. Detector directly consumes data from a C member without belonging to C, so B is Detector. U is Right and Monitor. While the transaction runs, local dispatch pauses for C, B and U. First, at the fence, C and B have pending candidates canceled and frozen sources retired. Then lifecycle callbacks run for C members only. If publication succeeds, it resets C and B selectors and advances their Tag generations. U keeps its Tag. If the transaction rolls back instead, Tags stay unchanged and canceled candidates are not replayed.
</desc>

<defs>
  <path d="M 0,0 L 8,4 L 0,8 Z" fill="currentColor" />
</defs>

Restart Left with a data-only Detector

<rect x="10" y="30" width="64" height="52" rx="8" />

<rect x="134" y="30" width="72" height="52" rx="8" />

<rect x="14" y="175" width="22" height="14" rx="3" />

<rect x="284" y="252" width="22" height="20" rx="4" />

<rect x="284" y="300" width="22" height="20" rx="4" />

<rect x="284" y="328" width="22" height="20" rx="4" />

<rect x="126" y="126" width="88" height="34" rx="8" />

<rect x="176" y="175" width="22" height="14" rx="3" />

<rect x="312" y="252" width="22" height="20" rx="4" />

<rect x="312" y="328" width="22" height="20" rx="4" />

<rect x="266" y="30" width="64" height="52" rx="8" />

<rect x="242" y="126" width="88" height="34" rx="8" />

<rect x="14" y="197" width="22" height="14" rx="3" />

<text x="42" y="61">
  Left
</text>

<text x="170" y="61">
  Fusion
</text>

<text x="298" y="61">
  Right
</text>

<text x="170" y="148">
  Detector
</text>

<text x="286" y="148">
  Monitor
</text>

<text x="104" y="43">
  Arm
</text>

<text x="236" y="43">
  Arm
</text>

<text x="104" y="80">
  data
</text>

<text x="236" y="80">
  data
</text>

<text x="178" y="108">
  data
</text>

<text x="42" y="186">
  C: Left, Fusion
</text>

<text x="204" y="186">
  B: Detector
</text>

<text x="42" y="208">
  U: Right, Monitor
</text>

Local dispatch pauses for C, B and U

<text x="30" y="266">
  Fence

  : cancel pending candidates
</text>

<text x="30" y="286">
  and retire frozen sources
</text>

<text x="30" y="314">
  Lifecycle

  : C members only
</text>

<text x="30" y="342">
  On success

  : reset selectors
</text>

<text x="30" y="362">
  and advance Tag generations
</text>

<text x="14" y="394">
  RolledBack

  : Tags unchanged, canceled
</text>

<text x="14" y="414">
  candidates are not replayed
</text>

<text x="295" y="267">
  C
</text>

<text x="323" y="267">
  B
</text>

<text x="295" y="315">
  C
</text>

<text x="295" y="343">
  C
</text>

<text x="323" y="343">
  B
</text>

Detector’s data edge adds boundary handling without adding lifecycle callbacks.

| Set and membership                                       | During the transaction                                                                         | On successful publication                            |
| -------------------------------------------------------- | ---------------------------------------------------------------------------------------------- | ---------------------------------------------------- |
| C: target and compiled callback dependents               | Cancel pending candidates and retire frozen sources at the fence, then run lifecycle callbacks | Reset selectors/readiness and advance Tag generation |
| B: direct same-process consumers of C outputs, outside C | Cancel pending candidates and retire frozen sources at the fence. No lifecycle callbacks       | Reset selectors/readiness and advance Tag generation |
| U: remaining operators                                   | Local dispatch pauses with the session. No lifecycle callbacks                                 | Retain Tag generation                                |

> **Warning**
>
> **Current boundary behavior.** B includes ordinary data consumers, not just
> explicitly authored channel boundaries. It is a direct-consumer expansion,
> not recursive traversal of every downstream operator. Its pending-work and
> generation handling may change in upcoming releases. Do not rely on B keeping
> its pending input selection or activation generation through a restart.

The runtime waits until every local owner reaches a control boundary with no
activation in progress. It then prepares the transaction, runs callbacks for C,
and resumes owners only after a resumable outcome. An unrelated operator is not
reconstructed, but uninterrupted progress is not promised. This is a local
session boundary, not distributed quiescence of interprocess neighbors.

Preparation completes before C's lifecycle callbacks:

| Order | Runtime work                                                                              |
| ----- | ----------------------------------------------------------------------------------------- |
| 1     | Check that C/B receivers are not draining end-of-stream and their completion remains open |
| 2     | Cancel and join the shared channel service, if used                                       |
| 3     | Cancel C/B pending candidates and retire their frozen sources                             |
| 4     | Begin the tensor restart when tensor resources exist, then prepare rate state             |

A failure here produces `kFailedStopped` and fails the run before any callback.
The channel service is shared across the session. Even a B receiver near
end-of-stream can fail the fence.

## Which stages repeat?

The executor uses the **effective** entry stage. Compatible queued requests can
coalesce into a union closure with the earliest shared entry stage. Several
receipts can therefore identify the same transaction.

| Effective entry | Pre-restart teardown | Forward suffix                                         |
| --------------- | -------------------- | ------------------------------------------------------ |
| `kStart`        | Stop                 | Start                                                  |
| `kArm`          | Stop                 | Arm → Start                                            |
| `kWarmUp`       | Stop                 | WarmUp → Arm → Start                                   |
| `kAllocate`     | Stop → Release       | Allocate → WarmUp → Arm → Start                        |
| `kDiscover`     | Stop → Release       | Discover → Allocate → WarmUp → Arm → Start             |
| `kConfigure`    | Stop → Release       | Configure → Discover → Allocate → WarmUp → Arm → Start |

A Start-entry transaction omits Release. That does not guarantee that Stop
preserves application resources: an operator may release resources in Stop and
recreate them in Start. Declare important teardown ordering at Stop/Release
explicitly. Ordinary teardown follows those stage graphs. Rollback unwinds
completed candidate work in reverse completion order. Independent Stop-peer
ordering in initial/final session transitions need not be the restart graph's
ordering.

## Choose the control operation

| Need                                                     | Operation                                           | Result                                                                   |
| -------------------------------------------------------- | --------------------------------------------------- | ------------------------------------------------------------------------ |
| Recover with current configuration                       | Explicit lifecycle restart                          | Re-enter the effective suffix in the same session                        |
| Change a runtime-mutable scalar                          | Parameter update                                    | Publish at an admitted next-compute boundary without a lifecycle restart |
| Change a lifecycle-mutable scalar                        | Parameter update plus lifecycle transaction         | Publish the candidate only after the transaction succeeds                |
| Replace topology, contracts or unsupported configuration | Compile an appropriate plan and start a new session | Fresh runtime instances and startup                                      |
| End execution                                            | Request stop and join                               | End this session                                                         |

Dedicated restart needs compile-time transaction capacity and explicit self,
external or requester-to-target grants. A restriction-only control handle cannot
add authority absent from the plan. Lifecycle-mutable parameter fields use
parameter-origin authority after admission. They do not require the dedicated
restart grant. Parameter control requires update/scope authority and its
own supported profile. Keep the restart and parameter profile matrices separate.

`compute()` can submit a nonblocking request and hand its receipt to off-callback
control code. Only control code outside runtime callbacks uses a
`LifecycleControlHandle` to inspect or await it. `LifecycleContext` cannot
submit or await restarts. Acceptance does not mean completion. A caller-side
wait timeout does not cancel the transaction.

See [Restart control](/holoscan/sdk-user-guide/advanced/restart-control) for the
complete APIs and [Runtime parameter updates](/holoscan/sdk-user-guide/advanced/runtime-parameter-updates)
for update-sequence/result observation. Parameter updates do not return dedicated
restart receipts. Atomic parameter batches are same-operator only: restarting
both cameras does not atomically change both cameras' parameter values.

## A model change has resource ownership

Consider an application-defined Detector with a scalar `uint32_t model_id` that
selects entries 0 and 1 in a fixed compatible catalog. Declare the field with
`.range(0U, 1U)` and `.lifecycle_mutable(holoscan::LifecycleStage::kAllocate)`.
Allocate owns model resources, WarmUp prepares them, Stop quiesces active use,
and Release destroys the owned resource set. This is a design example, not a
claim that the stock inference operator implements this contract.

For Detector's own update, C is `{Detector}` if it has no outgoing lifecycle or
rate dependencies. With no downstream consumer, B is empty. The incoming Fusion
data edge does not pull the cameras or Fusion into the closure. Detector was
in B for a Left restart. Membership depends on the target.

An out-of-range model ID is rejected with `ParamUpdateStatus::kOutOfBounds`
before lifecycle submission. To reason about restoration, instead use a valid
catalog ID whose Allocate succeeds but WarmUp fails:

1. Pre-restart Stop and Release use the old published value.
2. Candidate Allocate succeeds, but candidate WarmUp fails.
3. Candidate cleanup invokes Release for the completed allocation. The failed
   WarmUp callback must clean its own unrecorded partial acquisition with RAII.
4. RestoreOld reruns Allocate → WarmUp → Arm → Start using the old value.
5. If cleanup, restoration and runtime recovery succeed, the update rolls back
   and the run continues with the old published configuration.

Restoration loads the old resource again because teardown already released it.
There is no uninterrupted fallback instance or automatic reversal of external device effects.
Stock `InferenceOp` does not expose a lifecycle-mutable model-catalog selector.
Its Stop/Start restart reinitializes the same configuration. A different model
map or port contract needs supported reconfiguration or a new plan/session.

## Name the counter before asking whether it resets

| Identity                          | On `kPublished`                                                                                   | On `kRolledBack`                            |
| --------------------------------- | ------------------------------------------------------------------------------------------------- | ------------------------------------------- |
| `RunGenerationId`                 | Same RunSession                                                                                   | Same RunSession                             |
| `ParamGeneration`                 | Advances for each participating parameter candidate. A dedicated restart alone need not change it | Old published values and generations remain |
| `Tag.generation` / `Tag.sequence` | C/B generations advance and their activation sequences restart. U keeps its generation            | C/B generations remain unchanged            |
| `SampleMetadata::sequence`        | Runtime-owned output sequence continues across the restart                                        | Runtime-owned output sequence continues     |

External endpoints retain their own publication-sequence authority. These
counters describe different state and should be observed separately.

Pre-restart teardown sees old published parameters. Restart-forward sees the
candidate for a patched owner and retained current values for dependency-only
members. Cleanup sees the view that created the work. Restoration sees the old
view. Final teardown sees the then-current published view. Readiness, rate and
resource generations are further identities, not aliases for one universal epoch.

## Failure is not always recoverable

<title id="lid-outcome-title">
  Restart transaction outcomes: published, rolled back or fail-stopped
</title>

<desc id="lid-outcome-desc">
  A restart transaction after every owner has paused at the control boundary. Fence and runtime preparation come first. Then pre-restart teardown runs Stop, plus Release when the effective entry is Allocate or earlier. Then the candidate forward suffix runs. If it succeeds, runtime and candidate publication follow, and the result is kPublished: dispatch resumes, C and B Tag generations advance, and any parameter changes become published. If the forward suffix fails, completed candidate work is cleaned up, the old suffix is rerun with the old values and runtime services are recovered. The result is kRolledBack: the old lifecycle and values resume, Tags stay unchanged, a warning is reported and canceled work is not replayed. Fence and preparation, teardown, publication, cleanup, and restoration with recovery each carry a stop sign. If any of them fails, the result is kFailedStopped: the run fails, parameter candidates are retired, and partial publication may already have changed internal state.
</desc>

<defs>
  <path d="M 0,0 L 8,4 L 0,8 Z" fill="currentColor" />

  <path d="M 0,0 L 8,4 L 0,8 Z" fill="#5F9000" />
</defs>

Owners paused at the control boundary

<rect x="14" y="28" width="312" height="36" rx="8" />

<rect x="14" y="80" width="312" height="56" rx="8" />

<rect x="14" y="152" width="312" height="36" rx="8" />

<rect x="14" y="220" width="150" height="76" rx="8" />

<rect x="176" y="220" width="150" height="76" rx="8" />

<rect x="176" y="312" width="150" height="76" rx="8" />

<g fill="none" stroke="currentColor" />

<polygon points="319.2,42.2 319.2,49.8 313.8,55.2 306.2,55.2 300.8,49.8 300.8,42.2 306.2,36.8 313.8,36.8" />

<polygon points="319.2,94.2 319.2,101.8 313.8,107.2 306.2,107.2 300.8,101.8 300.8,94.2 306.2,88.8 313.8,88.8" />

<polygon points="157.2,234.2 157.2,241.8 151.8,247.2 144.2,247.2 138.8,241.8 138.8,234.2 144.2,228.8 151.8,228.8" />

<polygon points="319.2,234.2 319.2,241.8 313.8,247.2 306.2,247.2 300.8,241.8 300.8,234.2 306.2,228.8 313.8,228.8" />

<polygon points="319.2,326.2 319.2,333.8 313.8,339.2 306.2,339.2 300.8,333.8 300.8,326.2 306.2,320.8 313.8,320.8" />

<polygon points="33.2,539.2 33.2,546.8 27.8,552.2 20.2,552.2 14.8,546.8 14.8,539.2 20.2,533.8 27.8,533.8" />

<text x="26" y="51">
  Fence and runtime preparation
</text>

<text x="26" y="103">
  Teardown: Stop, plus Release when
</text>

<text x="26" y="123">
  the entry is Allocate or earlier
</text>

<text x="26" y="175">
  Candidate forward suffix
</text>

<text x="97" y="208">
  succeeds
</text>

<text x="259" y="208">
  fails
</text>

<text x="24" y="243">
  Runtime and
</text>

<text x="24" y="263">
  candidate
</text>

<text x="24" y="283">
  publication
</text>

<text x="186" y="243">
  Clean up
</text>

<text x="186" y="263">
  completed
</text>

<text x="186" y="283">
  candidate work
</text>

<text x="186" y="335">
  Rerun the old
</text>

<text x="186" y="355">
  suffix, recover
</text>

<text x="186" y="375">
  the runtime
</text>

<text x="14" y="362">
  Dispatch resumes
</text>

<text x="14" y="382">
  C/B Tags advance
</text>

<text x="14" y="402">
  Parameter changes
</text>

<text x="14" y="422">
  become published
</text>

<text x="176" y="454">
  Old lifecycle and
</text>

<text x="176" y="474">
  values resume
</text>

<text x="176" y="494">
  Tags unchanged
</text>

<text x="176" y="514">
  Warning, no replay
</text>

<text x="38" y="548">
  kFailedStopped

  : any marked step fails
</text>

<text x="14" y="570">
  The run fails. Parameter candidates are
</text>

<text x="14" y="590">
  retired, and partial publication may
</text>

<text x="14" y="610">
  already have changed internal state.
</text>

<text x="89" y="332">
  kPublished
</text>

<text x="251" y="424">
  kRolledBack
</text>

A stop sign marks a step whose failure ends the run. Restoration re-executes
the old suffix after completed candidate work has been cleaned up. It does not
undo arbitrary device effects. Only participating parameter candidates change
values on publication.

| Failure or outcome                                          | What follows                                                                                                        |
| ----------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------- |
| Fence/runtime preparation fails                             | FailedStopped before lifecycle callbacks                                                                            |
| Pre-restart teardown fails                                  | FailedStopped without a forward suffix or restoration                                                               |
| Forward callback fails                                      | Unwind completed candidate work, then attempt old suffix restoration if cleanup succeeds                            |
| Cleanup, restoration, runtime recovery or publication fails | FailedStopped and a failed run                                                                                      |
| `kPublished`                                                | Resume local dispatch. Publish participating parameter candidates and advance C/B receiver generations              |
| `kRolledBack`                                               | Continue with old lifecycle epoch and published parameters, unchanged C/B Tag generations, and a warning diagnostic |

RolledBack does not restore canceled candidates or queue contents. Its warning
carries `detail.transaction_id`. FailedStopped ends the run without relaunching
owners. A parameter candidate is retired without publication on that path.
Partial runtime publication may already have changed internal state, so a
failed run does not guarantee intact generations.

A stop request cancels queued work and closes ingress. An already
executing dedicated restart retains its actual outcome before terminal stop.
Parameter-origin work has an additional stop-before-publication fence.

## Inspect the plan, then observe the right result

| Question                                                | Evidence to inspect                                                           |
| ------------------------------------------------------- | ----------------------------------------------------------------------------- |
| Which checkpoints have callbacks?                       | `lifecycle_checkpoints()`, including `callback_present` and topological level |
| Which same-stage edges exist?                           | `lifecycle_dependencies()`                                                    |
| Who belongs to a target's closure?                      | `lifecycle_restart_closures()`                                                |
| Why was the entry stage widened?                        | Each closure record's `stage_resolutions` in `lifecycle_restart_closures()`   |
| Was a dedicated restart accepted and how did it finish? | Receipt, then control-handle `inspect()` / `await()`                          |
| Did a parameter candidate publish or roll back?         | Parameter control result for its accepted update sequence                     |
| Which activation generation was observed?               | Compare `ExecutionContext::tag()` before and after each transaction           |

The public plan does not expose every runtime receiver-binding record. For a
plain downstream consumer, derive the data relationship from the graph and
observe Tags and callback absence.
See [Plan tooling](/holoscan/sdk-user-guide/advanced/plan-tooling) for inspection.

For a restart that seems broader than expected, check forward lifecycle
reachability before the requested stage. Lifecycle-selected nominal rates can
add closure dependencies. A fixed-rate example does not exercise that path.
If resources need rebuilding earlier, declare the actual Allocate ordering or
lifecycle-mutable stage. Camera mode changes are device-specific: the owning
operator must quiesce acquisition, invalidate the right resources and restart
with valid timestamps. Restart does not promise physical exposure alignment.
Keep such reconfiguration outside a CUDA capture region, which rejects restart
targets or closure members.

## Check your prediction

Try answering these questions from the graph:

| Question                                       | Prediction                                   |
| ---------------------------------------------- | -------------------------------------------- |
| Can Left compute while Fusion is still in Arm? | No. Initial compute waits for global startup |
| Does restarting Left include Right?            | The baseline callback closure is Left/Fusion |
| What changes after adding the Stop edges?      | The closure includes both cameras and Fusion |
| Does a data-only Detector receive callbacks?   | No. It receives boundary handling            |

For a camera/IMU pipeline, predict the interval samples, missing policy and
end-of-stream behavior using the
[worked bracket](/holoscan/sdk-user-guide/advanced/advanced-temporal-patterns#bracketedwindow-all-or-nothing-neighborhoods).

## Recap

* Fixed stages provide shared ordering, restart suffixes and cleanup rules.
  Sparse hooks let each operator implement only its relevant work.
* Startup is global and stage-major. Restart callback scope is compiled from
  forward dependencies, then its effective entry stage is resolved.
* Local dispatch pauses at the control boundary. Callback members, direct
  boundary consumers and remaining operators have different restart effects.
* Publication, restoration and failed-stop outcomes preserve different state.
  Name each parameter, Tag and output counter explicitly.
* Lifecycle readiness, hardware synchronization and temporal data selection
  solve different problems. Declare each requirement through its own mechanism.