Tensors 101

So far the running example has moved small values between operators: IMU readings, control scalars. Camera frames are different — kilobytes to megabytes per sample, often living in GPU memory, produced at a fixed rate. holoscan::Tensor is the payload type built for exactly this: a cheap handle over runtime-owned, pool-backed storage, with CUDA stream ordering handled for you.

This page covers the basics you need for the first pipeline: declaring tensor ports, the producer hot path, why a received tensor “just works” on the GPU without any synchronization on your part, and how to reach the data from the host when you genuinely need to. The deeper machinery — pool sizing, the two-stream lifetime contract, foreign DLPack tensors, fanout — waits in Tensors in depth.

Complete C++ example: examples/tensors_101/cpp

Declaring tensor ports

Tensor ports follow the same three-declaration model as every operator: structure belongs in setup(). A TensorInputSpec groups consumer requirements, while a TensorOutputSpec groups producer promises, finite bounds, and storage intent. Both use TensorRepresentation for memory kind, dtype, rank, and stride facts. Omit the entire directional contract for an unconstrained pass-through port; a present contract must say something meaningful.

Here is the camera source and fusion consumer from the running example (camera_imu_control):

constexpr DLDataType kByteDtype{kDLUInt, 8U, 1U};
constexpr holoscan::TensorRepresentation kHostByteFrame{
    .memory_kind = holoscan::MemoryKind::kHost,
    .dtype = kByteDtype,
    .rank = 1U,
};
// CameraSource::setup() — the producer side
spec.output(frame, "frame")
    .max_emits_per_compute(1U)
    .produces_tensor(holoscan::TensorOutputSpec{
        .representation = kHostByteFrame,
        .bounds = holoscan::tensor_bounds(64U),
        .storage = holoscan::TensorOutputStorage::kRuntimePool,
    });
// WindowFusion::setup() — the consumer side
spec.input(camera, "camera")
    .queue_depth(4U)
    .expects_tensor(holoscan::TensorInputSpec{.representation = kHostByteFrame});

When both endpoints of an edge explicitly constrain the same layout field with incompatible values, compile() rejects the plan — one more check that runs before anything executes.

Storage intent and the shortest allocation form

An output defaults to pass-through storage. Calling allocate_tensor() without an admitted runtime pool returns kNotReady; allocating producers explicitly select TensorOutputStorage::kRuntimePool and finite bounds. The compiler derives the number of pool entries, so you do not construct a pool or count slots.

When the output declares an exact dtype, use frame.allocate_tensor(shape) instead of repeating it in TensorLoanRequest. For a single host fill, loan->write_host_and_commit(fill) combines writing and commit; then explicitly frame.emit(std::move(*loan)). Keep publication checked. The explicit lifecycle below explains the same operations; exhaustion and device-stream details are covered in Tensors in depth.

Producing a tensor: the hot path

Production is an explicit four-step lifecycle, and each step consumes the previous object. Every step returns holoscan::expected, so failures follow the same errors-as-values pattern as the rest of the SDK:

[[nodiscard]] holoscan::expected<void, holoscan::Error> compute(
    holoscan::ExecutionContext& ctx) override {
  // 1. Borrow a slot from the compiled pool.
  auto loan = frame.allocate_tensor(holoscan::TensorLoanRequest{
      .shape = kFrameShape, .dtype = kByteDtype});
  if (!loan) { return holoscan::make_unexpected(std::move(loan).error()); }
  // 2. Open a write phase on your compute stream (write_host() for host pools).
  auto writer = loan->write(ctx.cuda_stream());
  if (!writer) { return holoscan::make_unexpected(std::move(writer).error()); }
  // 3. Fill asynchronously on that same stream.
  cudaMemsetAsync(writer->data(), 0, writer->byte_size(), ctx.cuda_stream().get());
  // 4. Commit readiness, then publish.
  auto committed = std::move(*writer).commit();
  if (!committed) { return holoscan::make_unexpected(std::move(committed).error()); }
  return frame.emit(std::move(*loan),
                    holoscan::EmitOptions{.capture_time = ctx.activation_time()});
}

This device-pool form is demonstrated by the shipping tensors_in_depth example; tensors_101 and the camera example intentionally use host storage.

The move-only types enforce the order for you: the write guard must be committed (or destroyed) before the loan can be emitted, and every path consumes the loan exactly once. If you return early on an error, the loan and guard destructors abandon the allocation safely — nothing leaks, nothing half-publishes.

For the common “one callback fills the whole tensor” case, the fused helpers loan->write_and_commit(ctx.cuda_stream(), fill) and loan->write_host_and_commit(fill) collapse steps 2–4 into one call: the callback receives a const TensorOutputWriteGuard& to fill through, and commit happens on successful return. emit() stays a separate, explicit publication step.

The headline: stream ordering is automatic

This is the one fact to take away from this page. On the in-process data path, CUDA stream ordering between producer and consumer is the runtime’s job, not yours:

• On the producer side, commit() records your writes’ readiness on your compute stream. That is an asynchronous CUDA event record — it does not block the host, and it does not wait for your kernels to finish.
• On the consumer side, before compute() is invoked, the runtime enqueues an asynchronous wait on your compute stream for the producer’s readiness event. A received Tensor therefore arrives already ordered on your compute stream: any kernel you launch on ctx.cuda_stream() will observe the producer’s writes, even though neither of you ever synchronized.

You write, commit, and emit; the consumer receives and launches kernels. The GPU pipeline stays full end to end because nothing on the path ever blocks the host.

Receiving a tensor

A consumer declares expects_tensor(layout) and receives an ordinary sample:

auto frame = camera.receive();  // expected<Sample<Tensor>, Error>
if (!frame) { return holoscan::make_unexpected(std::move(frame).error()); }
const holoscan::Tensor& t = frame->data;
// Allocation-free metadata: t.shape_span(), t.dtype(), t.nbytes(), t.device().
// t is already ordered on this operator's compute stream — launch kernels on
// ctx.cuda_stream() that read t.data() directly.

The runtime keeps the backing storage alive and read-only for your whole activation; you do not manage leases or release anything. A received tensor is read-only by construction: data() is const, and mutable_data() returns ErrorCode::kPermissionDenied because you do not hold write authority.

A Tensor is a cheap handle

Copying a holoscan::Tensor copies a handle, not data: the copy retains one more claim on the same shared backing allocation, and zero payload bytes move. This is what makes tensors safe to pass around by value, stash in a small struct, or emit to several downstream operators — every recipient sees the same storage, and the pool recycles the slot only after the last claim retires. Treat a Tensor like a shared_ptr with tensor metadata attached, because that is essentially what it is.

Host access is explicit — and blocking

Everything above stays on the GPU. When you genuinely need bytes on the CPU — a verification tap, a file writer at the end of the pipeline — the tensor API makes you say so:

• t.synchronize_for_host() blocks the calling thread until the producer’s writes are observable from the host, then you may read t.data() directly (host-visible memory only).
• t.copy_to_host(dst, dst_bytes, stream) copies and blocks until the copy completes.
• t.copy_to_host_async(dst, dst_bytes, stream) enqueues the copy on your stream; you own the completion (for example, synchronize that stream in a sink operator that has nothing downstream).

Blocking is the honest cost of moving data to the host, so the API confines it to these clearly named calls. Keep them at the edges of your graph — never in a steady-state producer or a mid-pipeline transform.

The running example: camera frames as tensors

With this page, the camera leg of the camera + IMU pipeline becomes real: CameraSource borrows a frame-sized slot each activation, fills it, commits, and emits it with a capture timestamp; WindowFusion anchors its temporal window on the camera input, brackets the high-rate IMU samples around each frame, and reads frame metadata (frame->data.nbytes()) without ever copying the payload. The complete program is the camera_imu_control example application.

Going deeper

This page deliberately stopped at the in-process basics. When you need more, Tensors in depth covers the two-stream lifetime contract (readiness versus deallocation streams), how the compiler sizes pools, importing foreign DLPack tensors from libraries like CuPy or PyTorch (and why they need an explicit readiness stream), and multi-consumer fanout limits. To install PyTorch next to Holoscan, see Installing PyTorch.

Recap

• Declare Tensor structure in setup(): produces_tensor(TensorOutputSpec) on outputs and expects_tensor(TensorInputSpec) on inputs. Put finite bounds and kRuntimePool storage intent in the output spec.
• Produce with allocate_tensor() → write(ctx.cuda_stream()) → fill → commit() → emit(); the fused write_and_commit() covers the single-fill case.
• allocate_tensor() borrows a preallocated pool slot; exhaustion returns kResourceExhausted immediately — bounded by construction, no hidden malloc.
• Stream ordering is automatic: commit() records readiness without blocking, and a received tensor arrives already ordered on your compute stream. Never call cudaStreamSynchronize on the hot path.
• A Tensor is a cheap refcounted handle; copies share one backing, zero-copy.
• Host access is explicit and blocking (synchronize_for_host(), copy_to_host*) — keep it at the edges of the graph.

Next: with data flowing, step back and look at the full shape of an operator in Operator anatomy.