Lifecycle Callbacks
Most of an operator’s life is spent in compute() — the steady-state,
per-activation work. But some work belongs around steady state, not inside
it: validating immutable configuration, reserving a finite pool, warming up a
model, and later releasing everything cleanly. Lifecycle callbacks are where
that setup and teardown work lives, and the runtime drives them through a small
set of global stage barriers so the work is ordered correctly across every
operator in the graph during startup. A restart executes lifecycle callbacks
only for its compiled closure. Local dispatch pauses while the transaction runs.
This page shows the two ways to author that work — the simple
start()/stop() overrides and the advanced sparse stage callbacks — when
each hook actually runs, what an advanced callback can do through its
LifecycleContext, how to order lifecycle work across operators, what happens
when lifecycle work fails, and how restarts are authorized and requested.
For a worked stereo example and the reasons behind these rules, see Lifecycle and Restart in Depth.
Complete C++ example: examples/lifecycle_callbacks/cpp
Beta. Everything on this page works today as described: both authoring
paths, the stage barriers, cross-operator ordering, phase-selected parameter
views, and the rollback semantics. The API may still evolve in upcoming
releases. Not yet in baseline: automatic retry or backoff of a failed stage
(kRetryableFailure is a classification only), parallel execution of
independent same-level callbacks, and restart requests from inside a
lifecycle callback.
- The eight global lifecycle stages, from
kConfigurethroughkStartand the teardown pairkStop/kRelease - The two authoring paths — simple
start()/stop()overrides and sparsespec.lifecycle().stage(...)callbacks — and the one-callback-per-stage rule that connects them - When each hook actually runs, relative to construction,
setup(),compile(), the firstcompute(), shutdown, and restarts - Why advanced callbacks are
noexceptand return aLifecycleStatusvalue instead of throwing - What services
LifecycleContextexposes inside a callback - How to order lifecycle work across operators with dependency facts or
graph.lifecycle().before(...) - What happens when a lifecycle callback fails during startup or shutdown
- How deny-by-default restart permissions connect operator requests from
compute()and requests from external run control
The eight stage barriers
The runtime uses a fixed set of eight stages. During initial startup, every
operator completes a stage before any operator advances to the next stage.
A checkpoint without a callback completes as skipped and still participates
in ordering. Dependencies order operators within one stage. The barriers
already order different stages. The fixed stages also determine restart
suffixes and which completed work is eligible for Stop/Release cleanup. When a forward stage fails, the work completed in that pass is
rolled back in reverse completion order. Ordinary shutdown is not a blanket
reverse replay: it runs the compiled kStop and kRelease plans, in which
authored teardown dependencies take precedence and only independent same-level
kStop peers unwind in reverse of their successful start order.
Six forward (startup) stages run in this order:
Two teardown stages run on shutdown; the same pair also unwinds completed forward work when a startup pass or a restart’s forward suffix fails:
A requested completion or stop closes callback admission before teardown.
Callback claims ordered before that session-wide boundary may proceed and
retire; no claim ordered afterward enters. Teardown still runs, and a cleanup
failure overrides kCompleted, kRequestedCompleted, or kStopped with
terminal kFailed.
The helper is_lifecycle_forward_stage() returns true only for kConfigure
through kStart; the teardown stages are never valid restart entry points
(more on restarts below). An authorized restart re-enters the forward chain at
a compiler-resolved forward stage as a single transaction over its closure.
The requested entry can widen to an earlier stage within the closure. See
restart scope and entry.
Most operators register nothing — or just a matched kAllocate /
kRelease pair. Register a callback for a stage only when you genuinely have
work at that barrier.
Two authoring paths, one checkpoint model
Every operator has one logical checkpoint slot per stage, and two authoring paths can fill those slots. They meet in the same slots, which is what keeps the rules below simple.
The simple path: override start() and stop()
For plain setup and teardown at the start/stop boundary, override the public
virtuals Operator::start() and Operator::stop():
Graph authoring detects the overrides when the operator is added with
graph.op<T>(...) and lowers each one as a required callback in the same
checkpoint slot the advanced path uses: start() runs as your kStart
checkpoint, stop() as your kStop checkpoint. Your 4.x overrides port
unchanged (see
migrating from 4.x),
the no-argument void signatures remain valid, and nothing about the advanced
path deprecates them.
Inside start()/stop() the operator reads its parameters through
Operator::params(), which the runtime points at the exact phase-selected
generation before invoking the hook. Because stop() returns void and takes
no context, it cannot report a cleanup failure as a value (a thrown exception
is converted to kFatalFailure), and it cannot observe the cancellation
latch.
The advanced path: sparse stage callbacks
When work belongs at another stage (kAllocate, kArm, …), when you need
LifecycleContext services, or when stop()’s void return cannot express a
cleanup failure, register sparse callbacks per stage, inside setup(),
through the fluent spec.lifecycle() builder. An operator with no lifecycle
work registers nothing, and the runtime records a successful skipped
checkpoint for it at every barrier.
Each advanced callback is a member function with the exact signature
LifecycleStatus method(LifecycleContext&) noexcept. The builder takes a typed
member pointer, so the operator type is deduced — no string names:
.stage(...) registers a required callback; .optional(...) registers one
whose absence would have been permitted. If you need ordering within a
stage, express it across operators (see below) rather than stacking callbacks.
Registration itself is a cold authoring call: it can throw std::bad_alloc,
and an invalid stage, a duplicate stage, or a member pointer that does not
belong to the exact concrete operator type is reported when graph authoring
validates the declaration — not at the .stage(...) call itself.
At most one application callback per stage
The two paths meet at the same checkpoint slots, so at most one application
callback may fill a slot. An operator that overrides start() cannot also
register an advanced kStart callback (and likewise for stop()/kStop);
graph authoring rejects the conflict with GRAPH_LIFECYCLE_CALLBACK_DUPLICATE
when the operator is added. Mixing across stages is fine — advanced kStart
plus a plain stop() override is valid. At run time, Core invokes the one
callback that fills a slot, or records a successful skipped logical checkpoint
when neither path fills it. It never invokes an advanced callback and then the
virtual for the same stage.
When lifecycle stage callbacks actually run
Lifecycle stage callbacks — from kConfigure through kRelease, including
simple start()/stop() hooks lowered into kStart/kStop — run only during
RunSession lifecycle transitions. Operator construction, declaration, and
contract validation happen earlier and may be replayed:
- Graph authoring constructs a prototype.
graph.op<T>()constructs the authoring operator and calls itssetup()declaration method. - Compile replays declarations and contracts.
compile()constructs temporary factory instances and callssetup()andcontract()while validating and lowering the plan. An accepted parameter override can add another constructor/setup()replay for each affected operator. No lifecycle stage callback executes beforerun()/run_async(). - Each run constructs fresh operators. A
RunSessionrebuilds every runtime operator from its factory, callssetup(), and validates the replayedcontract()before any lifecycle stage callback. Treat constructors,setup(), andcontract()as deterministic, repeatable declaration and validation code: do not acquire external resources or perform externally visible work there. Acquire external resources in lifecycle callbacks instead.start()is not a constructor substitute, and ordinary operator-instance state from one run is not retained by the next. - Startup is stage-major. The six forward stages run one at a time behind
global barriers: every operator finishes stage N before any operator enters
stage N+1. Only after every operator’s
kStartcompletes does the runtime arm notifications, bind readiness sources, open control ingress, and enter the compute dispatch loop — in the initial startup of a run, the firstcompute()always happens after the lastkStart. - Callbacks run on the operator’s execution lane. Each callback body is
dispatched synchronously onto its operator’s assigned lane — not on the
thread that called
run()/run_async()— serially and in deterministic order: topological levels, then a stable registration-keyed order within a level. The baseline never runs two callbacks of one transition in parallel. - Shutdown fences inputs first. On completion,
request_stop(), or failure, the runtime first cancels readiness sources, closes notifications, stops channels, and closes tensor resources; only then does thekStopstage run, followed bykRelease, and finally the lanes stop. Advanced teardown callbacks can observectx.cancellation().stop_requested() == truewhen a stop has been requested; a simplestop()has no context parameter and cannot read the latch. - Restarts re-run hooks mid-run. A published restart tears down through
kStop(pluskReleasefor early re-entry stages) and re-runs the forward suffix, sostart()/stop()and advanced callbacks can run several times in one run. Restart teardown is not a cancellation request — the latch stays unset unless the run is actually stopping. See restart control.
Callback results: LifecycleStatus, not exceptions
Advanced lifecycle callbacks are noexcept and report their disposition as a
value, consistent with the SDK-wide errors-as-values model. (Simple
start()/stop() overrides return void; their failure story is in the next
section.) Supply an explanation with
ctx.report_error(error) or ctx.report_error(code, message) before returning a
failure status. Reporting copies bounded text without allocation or exceptions;
the first report wins for that invocation. It does not change the returned
status, and a successful callback does not publish a failure merely because it
reported detail. The next callback starts with fresh diagnostic state.
The four statuses:
Two sharp edges. First, each eligible callback is attempted once per
transition pass, with no automatic retry or backoff: kRetryableFailure
is a classification for future retry policy and is treated today like any
other failure of the pass. (A later admitted restart is a new transaction —
it can perform pre-restart teardown, a candidate forward attempt, rollback,
and old-state restoration, and your callback runs once in each phase it is
eligible for.) Second, a callback registered with .optional(...) that is
present and returns a failure still fails the transition — optional
means absence is permitted, not that failures are ignored.
When lifecycle work fails
The two paths fail differently, but both fail closed:
- Advanced callbacks cannot throw. They are
noexceptby signature — an exception that reaches the boundary terminates the process. Report failure as aLifecycleStatusvalue instead. - Simple
start()/stop()may throw. The runtime catches the exception and converts it tokFatalFailure; nothing escapesrun().
A failed startup rolls the run back and fails it:
- Work completed in the startup pass unwinds in reverse completion order —
kStopruns for operators that completedkArmorkStart, thenkReleasefor operators that completedkAllocate. - Simple
stop()follows a positional prefix rule during startup rollback: it runs only for operators that successfully completedkStartbefore the failing position in flattened start order. The failing operator’s ownstop()is skipped — even when itsstart()body was entered before throwing — and so is everything at or after its position. AdvancedkStopcallbacks are not subject to that cutoff; they keep ordinary rollback eligibility after a completedkArm. - The run reports
RunTermination::kFailedwith the diagnostic tokenRUNTIME_LIFECYCLE_CALLBACK_FAILED, and the later shutdown pass is a no-op —stop()is never invoked twice for one failure.
A failed kStop during ordinary shutdown never aborts cleanup: the
runtime keeps invoking every other eligible kStop, still runs kRelease,
and only then marks the run failed with the same token. See
understanding failures
for how the token surfaces in diagnostics().
Inside a callback: LifecycleContext
A callback receives a LifecycleContext& — a borrowed, callback-lifetime
facade. It exposes exactly the cold services a stage needs, and deliberately
nothing that could submit or await lifecycle work itself:
ctx.cancellation().stop_requested()— a cooperative stop / fatal-shutdown latch. In a forward callback, check it before beginning new work and returnkSkippedonly when no work was created. InkStoporkRelease, cancellation never removes the callback’s teardown obligations: perform every required quiescence and release action, using the latch only to omit optional work or shorten optional waits.ctx.params<Params>()— the complete immutable parameter aggregate selected for this callback phase. Repeated calls return the same address, and theParamstype must match the operator’s declared aggregate exactly.ctx.params_generation()— the phase-selected parameter generation number.ctx.transition_phase()— which transition phase selected this callback’s state view (startup, pre-restart teardown, restart forward, rollback, restore, or shutdown).ctx.resolve_clock(key)→expected<ClockRef, Error>— resolve a declared clock for this generation. Resolution is cold-only; no compute-time lookup is installed.ctx.restart_target(ref)→expected<OperatorRef, Error>— bind an already-authorized plan target to the current run generation. This resolves identity only; it does not submit a restart.
params<T>() in a callback is phase-selected. During a restart, the
parameter view a callback sees may be the candidate generation or the old
one — not necessarily the generation compute() reads. Read parameters fresh
inside each callback rather than caching a reference across stages. Simple
start()/stop() overrides read the same phase-selected view through
Operator::params(). See
typed parameters
for the parameter model itself.
Bind a lifecycle-selected output rate
An output whose exact nominal rate depends on device configuration or discovery can declare the stage that owns the selection:
bind_nominal_rate() accepts only an output owned by the callback’s Operator,
at the stage declared for that output, during startup or restart forward work.
The value must be a positive exact rational and cannot exceed
max_nominal_rate(). Repeating an equivalent normalized value in the same
candidate is idempotent; repeating a different value is a conflict. Ignoring a
failed bind does not make the candidate publishable.
The stage barrier is the visibility boundary. A same-stage ctx.rate() query
reports kPendingBarrier; a later stage sees the globally resolved value and
its propagated input/follows() dependents. During steady state,
ExecutionContext::rate(port) returns the same published generation in O(1)
without graph traversal or rate arithmetic. RunGenerationId stays stable for
the RunSession; RateGenerationId changes when startup or restart publishes
a new coherent rate snapshot.
For kConfigure and kDiscover, Core closes the selected barrier, evaluates
the complete queue/pool geometry, and performs framework-owned materialization
as the first kAllocate work, before any user kAllocate callback. A finite
maximum lets compilation preallocate the admitted ceiling and later install a
smaller logical active limit. An ordinary non-Tensor queue may instead remain
candidate geometry until this early barrier supplies the rate, because Core can
close and materialize that queue before user allocation begins.
Lifecycle-bound runtime-pool Tensor output is deliberately narrower. Without a finite maximum, only direct-host raw-Tensor geometry that Core can completely define is admitted. A descriptor-backed multi-field Tensor output needs a finite maximum and otherwise fails compilation without a partial allocation inventory. This Tensor exception must not be generalized into a requirement that every early lifecycle-bound non-Tensor queue declare a maximum.
kWarmUp and kArm occur after physical allocation. A rate selected there
therefore needs max_nominal_rate() or an exact explicit queue depth that
proves the preallocated domain. The later callback may change an active logical
limit within that domain, but it cannot allocate or silently grow backing.
On restart, re-enter at or before the declared binding stage to discover a new value. Compatible exact or larger backing can be retained while Runtime installs a fresh empty binding generation and a smaller active limit. If the candidate needs unadmitted growth or an incompatible provider, it fails before publication and the old generation remains recoverable. A restart that begins after the binding stage retains the published value rather than pretending the selection ran again.
The installed
rate_aware_ports example
runs this boundary end to end: Discover propagation and callback/compute
queries, in-capacity 100 Hz selections at both kWarmUp and kArm, typed
rejection of a 144 Hz selection above a 120 Hz maximum, and a 60-to-30 Hz
restart that retains the allocation generation while publishing a fresh rate
and storage-binding generation.
The following is Essential integration pseudocode, not a Core camera or receive-pool API. The bridge and its mode/layout types are vendor-owned:
Core owns rate admission and Core-owned queue/Tensor-pool materialization. An
HSB-shaped Essential adapter still needs independent maximum payload, layout,
alignment, placement, registration, descriptor, and in-flight bounds for its
own receive pool. A maximum frame rate bounds entry pressure; it does not prove
those byte or transport properties. During pre-restart teardown, use
transition_phase() to quiesce and retain adapter-owned backing. Compare the
newly discovered candidate with that retained geometry only after the new
Discover barrier; do not release a compatible larger pool merely because the
new mode is smaller.
Ordering across operators
The stage barriers already order work between stages. Within a stage, two declarative mechanisms order work across operators; the runtime lowers both into a deterministic dependency graph, so nothing depends on construction order.
Same-stage facts, declared per operator. An operator declares that its
checkpoint at a stage produces a named fact; another declares a
requires_dependency on that fact at a stage. The producer’s checkpoint then
completes before the requirer’s:
Direct order, declared at graph scope. From the containing graph, order two operators’ checkpoints in the same stage directly:
Prefer dependency facts when the operator itself knows what it provides or
needs (they travel with the operator and stay valid in any graph); use
before(...) when the ordering is a property of one particular composition.
Restart: graph permissions, operator and external requests
Restart authority is deny-by-default: nothing can restart an operator unless the containing graph granted a matching permission at authoring time.
An operator-originated restart is requested at run time from compute(),
through the ExecutionContext. External control code running outside Runtime
callbacks — such as an application supervisor — instead uses a restricted
LifecycleControlHandle, described on the
restart control page.
Handle submission from a Runtime callback is rejected, and a
LifecycleContext deliberately cannot submit through either path. These rules
keep callbacks from re-entering the lifecycle machinery that is running them:
A two-argument ctx.request_restart(target, options) overload requests the
restart of an authorized other operator. The from_stage must be a forward
stage (kConfigure through kStart) — kStop and kRelease are
teardown-only and can never be a restart entry point. The reason field in
LifecycleRestartOptions is a registered audit code, not free text.
Restart semantics — coalescing, rollback, transaction states, and external run control — have their own page: see restart control.
Putting it together
A Detector that reserves a pool at kAllocate, optionally warms up at
kWarmUp, and orders itself against an upstream operator’s readiness:
This listing is an illustrative variant. The shipping
lifecycle_callbacks example demonstrates the same sparse stage registration,
dependency ordering, optional kStart/kStop callbacks, and recorded
startup/teardown evidence.
For how DetectorParams declares its schema, see
parameter schemas.
Recap
- Startup and shutdown run through eight global stage barriers:
kConfigure→kDiscover→kAllocate→kWarmUp→kArm→kStart, withkStopandkReleasefor teardown. - Two authoring paths fill the same checkpoint slots: override
start()/stop()for the simple path, or register sparse advanced callbacks withspec.lifecycle().stage(...)/.optional(...). Every operator has one logical checkpoint per stage, and at most one application callback may occupy it. A same-stage conflict is rejected withGRAPH_LIFECYCLE_CALLBACK_DUPLICATE; an empty slot is a successful skipped checkpoint; and cross-stage mixing is valid. Most operators register nothing or justkAllocate/kRelease. - Lifecycle stage callbacks run only in
RunSessiontransitions. Authoring constructs a prototype and callssetup(); compile and eachRunSessionreconstruct operators and replaysetup()pluscontract()validation before lifecycle startup. EverykStartprecedes the firstcompute(), inputs are fenced beforekStop, andstop()/start()may repeat under restarts. - Advanced callbacks are
noexceptand returnLifecycleStatus;kSkippedis a distinct success; one attempt per eligible callback per transition pass, with no automatic retry; and a present optional callback that fails still fails the transition. A throwingstart()/stop()becomeskFatalFailure. - Startup rollback unwinds in reverse completion order and applies the
completed-
kStartprefix rule to simplestop(); akStopfailure during ordinary shutdown never aborts cleanup. LifecycleContextprovides cancellation, phase-selectedparams<T>(), clock resolution, and restart-target identity binding — read parameters fresh in every callback (simple hooks readOperator::params()instead).- Order within a stage with
produces/requires_dependencyfacts orgraph.lifecycle().before(...), never by construction order. - Restart permission is deny-by-default at graph scope. Operators request from
compute()viaExecutionContext::request_restart(...); external supervisors use a restrictedLifecycleControlHandle. Lifecycle callbacks cannot submit through either path.
Next: graph composition.