Understanding Failures

When something goes wrong in a Holoscan application, where it goes wrong tells you how it will be reported. The SDK follows the three-phase model — compose → compile → run — and each phase has exactly one failure style: authoring throws, compile() returns rejected plans, and the run fails closed. There is no fourth channel, and the styles never blur into each other: a wiring mistake never surfaces as a runtime crash, and a runtime resource problem never comes back as an exception from compile().

This page walks all three styles end to end with concrete examples, shows how to read diagnostic tokens (the debugging entry point for the compile and run phases), takes a closer look at kResourceExhausted — the signature failure of a bounded-by-construction runtime — and recaps error-code branching.

Complete C++ examples: examples/errors_as_values/cpp, examples/compile_and_check/cpp, and examples/running_and_stopping/cpp

One phase, one failure style

The split is deliberate. Composition happens on your development machine at the moment you write the bad line, so an exception that points straight at the call site is the fastest possible feedback. Compilation is a cold build step that tooling and CI want to inspect and diff, so it never unwinds — it hands back a fully queryable result even on rejection. And the run is where real work (and real hardware) is at stake, so the runtime refuses to guess: if a required binding or resource is unavailable, it stops before any of your callbacks run rather than silently degrading.

Style 1: authoring throws (compose time)

Graph composition APIs throw holoscan::RuntimeError when you wire something that cannot be wired — a stale handle, a flow whose source or target no longer exists, an invalid argument. This is a deliberate boundary choice, not a contradiction of the errors-as-values model: authoring mistakes are programmer errors at a construction boundary, and the throw lands exactly on the offending line.

#include <holoscan/core/graph.hpp>
holoscan::Graph graph{"pipeline"};
auto camera = graph.op<CameraOp>("camera");
auto fusion = graph.op<FusionOp>("fusion");
try {
  // A stale handle, a retired graph, or an invalid source/target throws
  // right here — not later at compile or run time.
  graph.add_flow(camera->frame, fusion->camera,
                 holoscan::ConnectionOptions{.queue_depth = 4U});
} catch (const holoscan::RuntimeError& e) {
  // e.code() recovers the ErrorCode (typically kInvalidArgument);
  // e.what() carries a stable token such as GRAPH_HANDLE_INVALID or
  // GRAPH_FLOW_SOURCE_INVALID plus context.
  log_error(e.what());
  throw;  // an authoring bug is not recoverable at this level
}

You normally do not write try/catch around composition at all: let the exception propagate and fix the code. The catch block above is for the rare host process (an editor plugin, a graph-building service) that composes graphs from untrusted descriptions and must survive a bad one.

Style 2: compile returns rejected plans

holoscan::compile() does not throw on admission failure. It returns a fully queryable rejected ExecutionPlan: plan.ok() is false, and plan.diagnostics() tells you why — as structured records, in canonical order, suitable for tooling and CI diffing. plan.require_no_errors() and attempting to run a rejected plan raise holoscan::CompileError. Exceptions from earlier composition, setup, or resource acquisition can propagate before a result exists; handle those at the application boundary while retaining their original type and explanation.

#include <holoscan/core/compile.hpp>
#include <holoscan/core/diagnostic.hpp>
#include <holoscan/core/execution_plan.hpp>
const holoscan::ExecutionPlan plan = holoscan::compile(graph);
holoscan::log_diagnostics(plan);  // Includes warnings even when plan.ok() is true.
if (!plan.ok()) {
  return;  // Never run a rejected plan.
}

Each Diagnostic carries a stable registry token, a severity (kWarning or kError — warnings preserve evidence without rejecting the plan), an optional bounded source locating the problem in the plan, and, where applicable, canonical required / provided summaries so a mismatch explains both sides of itself.

Tokens you are likely to meet at compile time:

• ENDPOINT_TYPE_MISMATCH — an exported or imported topic’s payload type does not match what the other side declares; check the required and provided fields on the record.
• PARTITION_LIMIT_EXCEEDED — the graph needs more execution partitions than the planner’s finite limit admits.
• PROFILE_CAPABILITY_UNSATISFIED — the requested profile demands a capability the composed graph or platform evidence cannot satisfy.
• ENDPOINT_NAME_GENERATED_PIN_REQUIRED — a boundary topic got a compiler-generated endpoint name that no pin acknowledges; under a constrained profile this is a hard error (author an explicit endpoint name, or pin the generated one).

When a record names a vertex or port, resolve it against plan.graph() — plan records use dense canonical IDs, not the authoring handles you held while composing. For the full compile workflow, see Compile and check.

Style 3: the run fails closed

A fatal invariant violation, such as calling a session wait from its own Runtime callback, terminates the process instead of producing a recoverable result. Its best-effort ERROR message bypasses the automatic-report opt-out, including API false and launch-time OFF, but respects normal logger filtering. Formatting or sink failure cannot prevent termination. There is no retained record or post-abort log_diagnostics() replay for these paths.

holoscan::run() returns 0 for natural kCompleted or explicit kRequestedCompleted; a compiler-accepted empty graph is one natural zero-work example. A finite source can also declare complete_after(ComputeCalls{N}): final-call publications are delivered and EOS drains downstream before natural completion. Successful callbacks, not emitted messages, are counted; an active unbounded source prevents whole-graph natural completion. With run_async(), those two states are successful, kStopped is an explicit stop, and kFailed is failure. Requested completion is immediate whole-run success, not EOS or downstream drain. Crucially, when a required runtime binding or resource is unavailable, the run terminates kFailed before any of your callbacks execute. The runtime never selects weaker semantics to limp along.

The only exceptions at this boundary are holoscan::CompileError for a default-constructed/no-state or rejected plan wrapper (check plan.ok() first) and std::system_error if the platform join fails. A plan produced by compiling an empty graph is distinct: when admitted, it completes as a zero-work run. Ordinary startup exhaustion, missing bindings, and execution failures arrive through the terminal state plus diagnostics:

#include <holoscan/core/diagnostic.hpp>
#include <holoscan/core/execution_plan.hpp>
#include <holoscan/core/run_session.hpp>
// Choose caller-owned presentation before launching.
holoscan::set_automatic_diagnostic_logging_enabled(false);
holoscan::RunSession session = holoscan::run_async(plan);
session.wait();
holoscan::log_diagnostics(session.diagnostics(), plan.graph());

Automatic reports default to enabled and need no helper call. The example opts out so its explicit terminal report does not duplicate the SDK summary; HOLOSCAN_AUTOMATIC_DIAGNOSTIC_LOGGING=ON overrides that API preference. run_async() returning does not mean startup is complete. The helper reports only the supplied prefix, without waiting, subscribing, draining, or replaying. Calling it after wait() gives it the complete retained terminal records. Inspection and repeated waits do not print; repeated helper calls may print again.

format_diagnostic() resolves primary and related endpoint identities against the immutable compiled graph and includes the diagnostic’s populated public fields. Pass the compiled graph from the diagnostic’s plan: its canonical IDs are distinct from source graph IDs. With an empty compiled graph view, numeric identities and owned names remain available. Related identities use related_graph_generation, related_graph, related_vertex_id, and related_port_id; related names prefer owned detail over graph lookup.

A primary port name copied into the detail appears once when it matches the compiled name. If the two nonempty names differ, the output preserves the compiled port_name and the owned captured_port_name. Use the rendered text for people, not program decisions: branch, grep, and assert on the structured token and fields rather than parsing the rendered line.

Original causes and operator locations

Callback failures include optional owning Diagnostic::detail evidence: the qualified operator path, operation, lifecycle stage and phase, original error code/message, and available nested causes. The normal failed-session ERROR report renders the same evidence; DEBUG logging is not required. Names and messages survive operator cleanup and destruction of the source graph.

For example, a start() exception explaining an unsupported image format is reported against the responsible operator’s kStart stage, rather than only a numeric lifecycle node. Tensor connection errors identify both ports and the incompatible fields with provided and required values. Core preserves the module’s explanation; it does not infer image-format semantics itself.

Concrete Tensor output is validated again when it is emitted. If an output is declared as cuda_device but compute emits a host Tensor and propagates that rejection, the callback diagnostic names the output and records memory_kind, provided host, and required cuda_device. The first rejected output in the callback supplies this supplemental context. Handling the rejection and returning success does not emit a failure report; a later activation starts with fresh context. If that same callback returns a different Error, its original code and message remain the primary error.

A valid partition’s callback failure is reported as an operator failure. RUNTIME_COMPUTE_DISPATCH_FAILED instead identifies a dispatch rejection before callback entry and retains its original Error. RUNTIME_COMPUTE_CUDA_DEVICE_RESTORE_FAILED identifies failed device restoration after a callback: it is a separate runtime record, linked to that operator’s failed callback when present. If compute succeeded, restoration is the primary failure. Genuine nested exceptions stay in detail.causes; cleanup failures are separate records.

Rollback, restoration, and teardown errors have distinct roles. When a later failure refers to an earlier retained primary record, detail.primary_diagnostic_index is a zero-based index in that report, not a registry DiagnosticId. Restart failures also retain detail.transaction_id, which can be correlated with lifecycle receipts. A failed restart that restores the previous generation remains warning evidence rather than a terminal run failure. Report formatting puts errors before warnings while retaining their relative order; causal indices still refer to the original diagnostic sequence. Runtime resource cleanup, including CUDA quiesce and shutdown, uses the teardown role even when it is not a lifecycle callback. A cleanup failure with no earlier retained terminal failure has no primary diagnostic index.

Cause capture is bounded to 4096 bytes and four cause entries. Truncation or unavailable optional detail is explicit. Session diagnostics retain at most sixteen records, reserving space for the first terminal failure and an overflow marker; overflow from advisory records may become visible only at finalization. The overflow marker is a warning when only warnings were lost, even if a separate terminal error was retained. Losing an error record produces an error-severity marker. Advisory overflow alone does not fail a run. format_diagnostics(records, graph) renders a report capped at 16384 bytes. Terminal control characters in messages are escaped. Do not include secrets in exception or error messages.

Supplemental detail is available through the in-memory APIs and formatters; it is not serialized in canonical execution-plan JSON and does not affect the plan digest. Stable tokens and existing structural fields remain the machine-readable classification. Adding the public detail field requires consumers to rebuild against the updated SDK, without changing operator callback signatures.

Runtime tokens you are likely to meet:

• RUNTIME_CLOCK_BINDING_UNAVAILABLE — the plan requires a clock the runtime cannot bind in this process; check that every clock named in the plan’s clock catalog is available where you run.
• RUNTIME_PARAMETER_BINDING_UNAVAILABLE — operators declare typed parameters but the parameter binding the plan expects is not installed.
• RUNTIME_OPERATOR_REPLAY_MISMATCH — an operator reconstructed for this run does not match what the plan recorded at compile time; recompile after changing operator code rather than reusing a stale serialized plan.
• RUNTIME_LIFECYCLE_CALLBACK_FAILED — one of your lifecycle callbacks (a start()/stop() override or an advanced stage callback) reported failure or threw, failing the affected scope closed. During startup this means the run rolled back — completed work unwound in reverse, with simple stop() running only for the operators that completed kStart earlier; during shutdown, cleanup still ran to completion before the run was marked failed. Cleanup failure overrides natural or requested success and stop. See lifecycle callbacks for the exact rollback rules.
• RUNTIME_MANAGED_SIGNAL_RECEIVED — a first synchronously accepted SIGINT or SIGTERM requested cooperative stop. This is warning evidence on a clean kStopped session, not a failure; a direct application request_stop() does not add it.
• RUNTIME_OPERATOR_PUBLICATION_STOPPED — an operator’s callback failed only because an accepted stop cut one of its publications short: the recipient could not accept, and callback admission had closed. This is warning evidence on a kStopped session, not a failure. See publications blocked at stop.
• RUNTIME_MANAGED_SIGNAL_DOMAIN_REQUIRED — default Linux run(plan) had no adoptable domain and a successful census read observed anything other than exactly one live process task, or the successful second read changed from the first. Create an explicit domain before any background threads, or deliberately select RunSignalPolicy::kCallerManaged.
• RUNTIME_MANAGED_SIGNAL_OWNER_BUSY — an existing domain or binding is unavailable, or the process registry is poisoned. Let an active binding retire, close and recreate a spent domain, or reconcile signal ownership. After registry poisoning, contain and restart the process rather than retrying.
• RUNTIME_MANAGED_SIGNAL_INSTALL_FAILED — mask, census read, disposition, descriptor, broker, or binding setup failed before a usable managed run was established. The typed detail identifies the exact setup phase.
• RUNTIME_MANAGED_SIGNAL_WAIT_FAILED — synchronous signal waiting, consumption, or controller notification failed. An attached session stops and terminates kFailed after cleanup; later managed admission remains poisoned.
• RUNTIME_MANAGED_SIGNAL_RESTORE_FAILED — delegation, descriptor cleanup, disposition validation, or exact creator-mask restoration failed. Treat the process as contaminated and restart it rather than assuming later managed admission is safe.
• RUNTIME_MANAGED_SIGNAL_ESCALATED — warning evidence that a second accepted occurrence was delegated under the unchanged prior disposition. It does not replace that disposition’s behavior.

session.diagnostics() is append-only and stable: the span you hold stays valid, and a later call may expose a longer prefix. Until the session is terminal, do not treat the prefix you observed as complete — call it again.

Tokens are the debugging entry point

Across compile and run, the Diagnostic token is where debugging starts. It is a stable registry spelling, not free-form log prose: the registry is append-only, so a token you branch on or grep for today keeps its meaning across SDK releases. That makes tokens the right key for CI assertions (“this graph must not produce ENDPOINT_NAME_GENERATED_PIN_RECOMMENDED”), for dashboards, and for your own triage notes. Never match on rendered message text — wording can improve; tokens do not change.

A practical triage loop:

1. Read the token — its spelling names the subsystem (GRAPH_*, ENDPOINT_*, PARTITION_*, PROFILE_*, RUNTIME_*) and the condition.
2. Check severity — a kWarning preserves evidence without rejecting or failing; only kError records reject a plan or fail a run.
3. Use source (and required / provided when present) to locate the vertex or port, resolving IDs against plan.graph().
4. Follow cause_id when set — it points at the earlier record that caused this one, so fix the root record first.

kResourceExhausted: the bounded-runtime signature

Holoscan is bounded by construction: payload pools are preallocated, queues have finite depth, and fanout tracking has fixed capacity. Nothing on the hot path allocates on demand, so when a bound is hit the SDK does not grow a buffer or block indefinitely — it returns holoscan::ErrorCode::kResourceExhausted. Seeing it is not a runtime bug; it is the runtime telling you a capacity you configured (or defaulted) does not match your workload. Check, in order:

• Pool bounds. Output<Tensor>::allocate_tensor() borrows a slot from a preallocated pool; when every slot is on loan, allocation returns kResourceExhausted. If a downstream consumer holds tensors longer than you sized for, grow the pool — or release sooner. See Tensors in depth.
• Queue depths. Edge queues are sized at composition via ConnectionOptions::queue_depth. Persistent exhaustion or backpressure on an edge means the consumer cannot keep up with the producer at that depth — fix the rate mismatch or deepen the queue deliberately.
• Fanout caps. Shared payloads track their consumers against fixed capacities (for example, the bounded fanout release-stream count on GPU-backed buffers). A one-to-many edge with more simultaneous borrowers than the cap admits exhausts it.

Its transient sibling is kBackpressured — a channel or queue momentarily full. Backpressure is a normal flow-control outcome you retry or shed; kResourceExhausted from a pool means a sizing decision needs revisiting.

Backpressure is about simultaneous retention, not just average rate

This behavior is not specific to borrowed inputs or Tensor output loans. Input access reads a selection that Runtime froze before compute(); neither owning receive() nor borrowed selected_sample() pops and frees a live queue position. Runtime protects the selected input through callback retirement. Owning read claims, temporal retention, and asynchronous Tensor work can have additional lifetime costs beyond that callback.

A producer that is faster than its consumer for a sustained interval will fill a finite lossless queue. But even a slower producer can encounter transient pressure from bursts, scheduling jitter, a slow fanout recipient, or overlap with a still-active consumer. For example, with separate producer and consumer partitions and a depth-one direct blocking edge:

1. The consumer processes message 1 in its sole typed receive slot.
2. It signals “processed” from inside compute(), before Runtime retires the selected input.
3. Another partition publishes message 2 before that retirement completes.
4. Publication returns kBackpressured; overwriting message 1 would violate the active consumer’s lifetime guarantee.

This can happen with only one new message following each acknowledgment, even when average processing capacity is ample. Finishing a read, releasing a local view, or setting an application flag is not proof that Runtime has retired the selection. Transport buffering may absorb this overlap in one configuration; do not depend on that extra staging or change partitions to conceal a missing flow-control guarantee.

Size capacity and choose the pressure policy explicitly

• Size for a bounded burst and retention interval. Count simultaneous unread and active/retained work, not just average messages per second. On a direct blocking edge, the active selection occupies typed delivery capacity: one active message plus one additional admitted message requires room for that overlap. Depth two covers that particular bound, not arbitrary workloads. Temporal windows, fanout and retained Tensor backing also affect their own compiler-derived storage bounds. Inspect the accepted plan and Tensor sizing guidance rather than assuming queue depth alone is the Tensor pool size.
• No finite queue fixes sustained overload. Reduce the admitted source rate, increase processing capacity, or deliberately permit loss. Increasing depth trades memory and latency for tolerance of a finite burst or stall.
• For lossy processing, make loss intentional. Select kDropOldest when replacing unread samples matches the application contract, or explicitly count and drop a pre-submit rejected output. Lossy queues still must not overwrite active borrowed data or unresolved Tensor backing.
• For lossless processing, preserve work and admission authority. Use a proven in-flight bound or credits that reflect actual available capacity. Explicit deferred retry needs bounded owning pending state, a guaranteed retry trigger independent of new input, and upstream admission control. Merely returning success on pressure does not preserve an OnEach input for later processing. There is no general automatic lossless callback-retry API.

Error-code branching recap

Inside operators and application glue, fallible calls return holoscan::expected<T, Error>; you branch on Error::code, never on message text — codes are the stable identity, messages are human context:

For example, this fragment inside compute() deliberately drops a rejected frame; dropped_frames_ is an operator-owned counter. It is not a lossless retry implementation:

auto result = output_.emit(std::move(frame));
if (!result) {
  if (result.error().code == holoscan::ErrorCode::kBackpressured) {
    ++dropped_frames_;
    return {};  // intentional loss; this input will not be retried
  }
  return holoscan::make_unexpected(std::move(result).error());
}
return {};

runtime_error_message(error) renders "<code name>: <message>" for logs, and error_code_name(code) gives the enumerator spelling. Code-only errors allocate nothing, which is why hot paths return them freely. For the full value-channel model, including the HOLOSCAN_TRY propagation macros, see Errors as values.

Recap

• One failure style per phase: compose throws RuntimeError, compile() returns a rejected plan, the run fails closed with terminal kFailed — and nothing crosses between the styles.
• Check plan.ok() after every compile and iterate plan.diagnostics() on rejection; use require_no_errors() only when downstream setup cannot proceed without an admitted plan.
• rc == 0 means natural kCompleted or explicit kRequestedCompleted; requested completion is immediate success without EOS or drain guarantees. Inspect session.diagnostics() on kFailed, and re-poll it until terminal.
• Diagnostic tokens are stable registry spellings — branch, grep, and assert on tokens, never on message text.
• kResourceExhausted means a configured bound is undersized: check pool bounds, queue depths, and fanout caps; kBackpressured is its transient, retryable sibling.
• Next: capture the full picture with structured Logging.