> This page is for version 5.0 (Latest) (default). > For other versions, use one of these documentation indexes: > - 5.0 (Latest) (default): https://docs.nvidia.com/holoscan/sdk-user-guide/5-0-latest/llms.txt > - ltsb-2.0: https://docs.nvidia.com/holoscan/sdk-user-guide/ltsb-2-0/llms.txt > - 4.6: https://docs.nvidia.com/holoscan/sdk-user-guide/4-6/llms.txt > - 4.5: https://docs.nvidia.com/holoscan/sdk-user-guide/4-5/llms.txt > - 4.4: https://docs.nvidia.com/holoscan/sdk-user-guide/4-4/llms.txt > - 4.3: https://docs.nvidia.com/holoscan/sdk-user-guide/4-3/llms.txt > - Archives: https://docs.nvidia.com/holoscan/sdk-user-guide/archives/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/holoscan/sdk-user-guide/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/holoscan/sdk-user-guide/_mcp/server. So far the running example has moved small values between operators: IMU readings, control scalars. Camera frames are different — kilobytes to megabytes per sample, often living in GPU memory, produced at a fixed rate. `holoscan::Tensor` is the payload type built for exactly this: a cheap handle over runtime-owned, pool-backed storage, with CUDA stream ordering handled for you. This page covers the basics you need for the [first pipeline](/holoscan/sdk-user-guide/getting-started/first-pipeline): declaring tensor ports, the producer hot path, why a received tensor "just works" on the GPU without any synchronization on your part, and how to reach the data from the host when you genuinely need to. The deeper machinery — pool sizing, the two-stream lifetime contract, foreign DLPack tensors, fanout — waits in [Tensors in depth](/holoscan/sdk-user-guide/data-and-payloads/tensors-in-depth). **Complete C++ example:** [`examples/tensors_101/cpp`](https://github.com/nvidia-holoscan/holoscan-sdk/tree/v5.0.0/examples/tensors_101/cpp) > **You will learn** > > * How to declare Tensor ports with direction-specific > `produces_tensor()` and `expects_tensor()` contracts > * The producer lifecycle: `allocate_tensor()` → `write()` → fill → `commit()` → > `emit()` > * Why stream ordering is automatic — and why you must never call > `cudaStreamSynchronize` on the hot path > * Why `allocate_tensor()` borrows from a pool instead of allocating, and what > `kResourceExhausted` means > * How to copy `Tensor` handles cheaply and access data from the host explicitly ## Declaring tensor ports Tensor ports follow the same three-declaration model as every operator: structure belongs in `setup()`. A `TensorInputSpec` groups consumer requirements, while a `TensorOutputSpec` groups producer promises, finite bounds, and storage intent. Both use `TensorRepresentation` for memory kind, dtype, rank, and stride facts. Omit the entire directional contract for an unconstrained pass-through port; a present contract must say something meaningful. Here is the camera source and fusion consumer from the running example (`camera_imu_control`): ```cpp constexpr DLDataType kByteDtype{kDLUInt, 8U, 1U}; constexpr holoscan::TensorRepresentation kHostByteFrame{ .memory_kind = holoscan::MemoryKind::kHost, .dtype = kByteDtype, .rank = 1U, }; // CameraSource::setup() — the producer side spec.output(frame, "frame") .max_emits_per_compute(1U) .produces_tensor(holoscan::TensorOutputSpec{ .representation = kHostByteFrame, .bounds = holoscan::tensor_bounds(64U), .storage = holoscan::TensorOutputStorage::kRuntimePool, }); // WindowFusion::setup() — the consumer side spec.input(camera, "camera") .queue_depth(4U) .expects_tensor(holoscan::TensorInputSpec{.representation = kHostByteFrame}); ``` When both endpoints of an edge explicitly constrain the same layout field with incompatible values, `compile()` rejects the plan — one more check that runs [before anything executes](/holoscan/sdk-user-guide/getting-started/compile-and-check). > **Warning** > > A runtime-pool contract needs finite metadata and byte capacities. Exact rank > supplies the metadata capacity here; `tensor_bounds(64U)` supplies the byte cap. ## Storage intent and the shortest allocation form An output defaults to pass-through storage. Calling `allocate_tensor()` without an admitted runtime pool returns `kNotReady`; allocating producers explicitly select `TensorOutputStorage::kRuntimePool` and finite bounds. The compiler derives the number of pool entries, so you do not construct a pool or count slots. When the output declares an exact dtype, use `frame.allocate_tensor(shape)` instead of repeating it in `TensorLoanRequest`. For a single host fill, `loan->write_host_and_commit(fill)` combines writing and commit; then explicitly `frame.emit(std::move(*loan))`. Keep publication checked. The explicit lifecycle below explains the same operations; exhaustion and device-stream details are covered in [Tensors in depth](/holoscan/sdk-user-guide/data-and-payloads/tensors-in-depth). ## Producing a tensor: the hot path Production is an explicit four-step lifecycle, and each step consumes the previous object. Every step returns `holoscan::expected`, so failures follow the same [errors-as-values](/holoscan/sdk-user-guide/getting-started/errors-as-values) pattern as the rest of the SDK: ```cpp [[nodiscard]] holoscan::expected compute( holoscan::ExecutionContext& ctx) override { // 1. Borrow a slot from the compiled pool. auto loan = frame.allocate_tensor(holoscan::TensorLoanRequest{ .shape = kFrameShape, .dtype = kByteDtype}); if (!loan) { return holoscan::make_unexpected(std::move(loan).error()); } // 2. Open a write phase on your compute stream (write_host() for host pools). auto writer = loan->write(ctx.cuda_stream()); if (!writer) { return holoscan::make_unexpected(std::move(writer).error()); } // 3. Fill asynchronously on that same stream. cudaMemsetAsync(writer->data(), 0, writer->byte_size(), ctx.cuda_stream().get()); // 4. Commit readiness, then publish. auto committed = std::move(*writer).commit(); if (!committed) { return holoscan::make_unexpected(std::move(committed).error()); } return frame.emit(std::move(*loan), holoscan::EmitOptions{.capture_time = ctx.activation_time()}); } ``` This device-pool form is demonstrated by the shipping `tensors_in_depth` example; `tensors_101` and the camera example intentionally use host storage. Producer tensor loan lifecycle: borrow a pool slot, write on the stream, commit readiness, emit to the consumer Tensor pool fixed slots, sized at compile() out borrow, no malloc allocate_tensor() loan write(stream) guard, fill async commit() records readiness no host sync emit(loan) publish Consumer receive(): already stream-ordered The move-only types enforce the order for you: the write guard must be committed (or destroyed) before the loan can be emitted, and every path consumes the loan exactly once. If you return early on an error, the loan and guard destructors abandon the allocation safely — nothing leaks, nothing half-publishes. For the common "one callback fills the whole tensor" case, the fused helpers `loan->write_and_commit(ctx.cuda_stream(), fill)` and `loan->write_host_and_commit(fill)` collapse steps 2–4 into one call: the callback receives a `const TensorOutputWriteGuard&` to fill through, and commit happens on successful return. `emit()` stays a separate, explicit publication step. > **Why is it called allocate_tensor()?** > > Despite the name, `allocate_tensor()` never allocates memory on the hot path. > The pool behind your output is fully materialized before the graph starts > running, sized from your declared bounds and the graph's queue and fanout > capacities. The call **borrows** one preallocated slot and hands you a > `TensorOutputLoan`. If every slot is checked out, it returns > `ErrorCode::kResourceExhausted` immediately — it never falls back to `malloc`, > never grows the pool, and never waits. That is the SDK's *bounded by > construction* principle applied to bulk data: memory use is fixed at compile > time, and exhaustion is a typed value you can branch on, not a stall. ## The headline: stream ordering is automatic This is the one fact to take away from this page. On the in-process data path, CUDA stream ordering between producer and consumer is the **runtime's job, not yours**: * **On the producer side**, `commit()` records your writes' readiness on your compute stream. That is an asynchronous CUDA event record — it does **not** block the host, and it does not wait for your kernels to finish. * **On the consumer side**, before `compute()` is invoked, the runtime enqueues an asynchronous wait on *your* compute stream for the producer's readiness event. A received `Tensor` therefore arrives **already ordered on your compute stream**: any kernel you launch on `ctx.cuda_stream()` will observe the producer's writes, even though neither of you ever synchronized. You write, commit, and emit; the consumer receives and launches kernels. The GPU pipeline stays full end to end because nothing on the path ever blocks the host. > **Warning** > > **Never call `cudaStreamSynchronize` on the hot path.** `commit()` records > readiness; it is not a synchronize, and it does not need one. Adding a blocking > sync "just to be safe" serializes the CPU against the GPU, collapses the > pipelining the runtime built for you, and buys no additional correctness. The > only blocking calls in the tensor API are the explicit host-access methods > described below — and those do not belong in a steady-state `compute()` either. ## Receiving a tensor A consumer declares `expects_tensor(layout)` and receives an ordinary sample: ```cpp auto frame = camera.receive(); // expected, Error> if (!frame) { return holoscan::make_unexpected(std::move(frame).error()); } const holoscan::Tensor& t = frame->data; // Allocation-free metadata: t.shape_span(), t.dtype(), t.nbytes(), t.device(). // t is already ordered on this operator's compute stream — launch kernels on // ctx.cuda_stream() that read t.data() directly. ``` The runtime keeps the backing storage alive and read-only for your whole activation; you do not manage leases or release anything. A received tensor is read-only by construction: `data()` is `const`, and `mutable_data()` returns `ErrorCode::kPermissionDenied` because you do not hold write authority. ## A Tensor is a cheap handle Copying a `holoscan::Tensor` copies a *handle*, not data: the copy retains one more claim on the same shared backing allocation, and zero payload bytes move. This is what makes tensors safe to pass around by value, stash in a small struct, or emit to several downstream operators — every recipient sees the same storage, and the pool recycles the slot only after the last claim retires. Treat a `Tensor` like a `shared_ptr` with tensor metadata attached, because that is essentially what it is. ## Host access is explicit — and blocking Everything above stays on the GPU. When you genuinely need bytes on the CPU — a verification tap, a file writer at the end of the pipeline — the tensor API makes you say so: * `t.synchronize_for_host()` blocks the calling thread until the producer's writes are observable from the host, then you may read `t.data()` directly (host-visible memory only). * `t.copy_to_host(dst, dst_bytes, stream)` copies and blocks until the copy completes. * `t.copy_to_host_async(dst, dst_bytes, stream)` enqueues the copy on your stream; you own the completion (for example, synchronize that stream in a sink operator that has nothing downstream). Blocking is the honest cost of moving data to the host, so the API confines it to these clearly named calls. Keep them at the edges of your graph — never in a steady-state producer or a mid-pipeline transform. ## The running example: camera frames as tensors With this page, the camera leg of the [camera + IMU pipeline](/holoscan/sdk-user-guide/examples/camera-imu-control) becomes real: `CameraSource` borrows a frame-sized slot each activation, fills it, commits, and emits it with a capture timestamp; `WindowFusion` anchors its temporal window on the camera input, brackets the high-rate IMU samples around each frame, and reads frame metadata (`frame->data.nbytes()`) without ever copying the payload. The complete program is the `camera_imu_control` example application. ## Going deeper This page deliberately stopped at the in-process basics. When you need more, [Tensors in depth](/holoscan/sdk-user-guide/data-and-payloads/tensors-in-depth) covers the two-stream lifetime contract (readiness versus deallocation streams), how the compiler sizes pools, importing foreign DLPack tensors from libraries like CuPy or PyTorch (and why they need an explicit readiness stream), and multi-consumer fanout limits. To install PyTorch next to Holoscan, see [Installing PyTorch](/holoscan/sdk-user-guide/setup/additional-setup/pytorch). ## Recap * Declare Tensor structure in `setup()`: `produces_tensor(TensorOutputSpec)` on outputs and `expects_tensor(TensorInputSpec)` on inputs. Put finite bounds and `kRuntimePool` storage intent in the output spec. * Produce with `allocate_tensor()` → `write(ctx.cuda_stream())` → fill → `commit()` → `emit()`; the fused `write_and_commit()` covers the single-fill case. * `allocate_tensor()` borrows a preallocated pool slot; exhaustion returns `kResourceExhausted` immediately — bounded by construction, no hidden `malloc`. * Stream ordering is automatic: `commit()` records readiness without blocking, and a received tensor arrives already ordered on your compute stream. Never call `cudaStreamSynchronize` on the hot path. * A `Tensor` is a cheap refcounted handle; copies share one backing, zero-copy. * Host access is explicit and blocking (`synchronize_for_host()`, `copy_to_host*`) — keep it at the edges of the graph. Next: with data flowing, step back and look at the full shape of an operator in [Operator anatomy](/holoscan/sdk-user-guide/building-operators/operator-anatomy).