Instrument Applications

Adding Scopes and Marks

View as Markdown

Use this guide when you want NeMo Relay to identify one agent run, request, workflow, or operation before you instrument individual tool and LLM calls.

What You Build

You will create an active scope, emit named mark events inside that scope, and validate that subscribers can observe the lifecycle. The result is a small trace boundary that later tool calls, LLM calls, middleware, and exporters can attach to.

Scopes give emitted work ownership. Marks record point-in-time checkpoints inside that ownership boundary.

Before You Start

Complete one binding Quick Start guide first:

You should know which application boundary should own the trace. Common scope boundaries include:

  • One HTTP request
  • One agent run
  • One workflow step
  • One background job
  • One tenant-isolated experiment

Integration Pattern

Follow this sequence to keep framework work attached to the expected runtime context.

  1. Choose the scope boundary that owns the work.
  2. Register a temporary subscriber while validating instrumentation.
  3. Push or enter the scope before tool and LLM work begins.
  4. Emit marks for important checkpoints that are not full nested calls.
  5. Close the scope when the request, agent run, or workflow ends.
  6. Confirm that the subscriber sees scope start, mark, and scope end events.

Minimal Example

The examples below create one agent-run scope and emit two marks.

1import nemo_relay
2
3def log_event(event) -> None:
4    print(f"{event.kind} {event.name}")
5
6nemo_relay.subscribers.register("scope-check", log_event)
7
8try:
9    with nemo_relay.scope.scope(
10        "agent-run",
11        nemo_relay.ScopeType.Agent,
12        input={"request_id": "req-123"},
13    ) as handle:
14        nemo_relay.scope.event("planning-started", handle=handle, data={"step": 1})
15        nemo_relay.scope.event("planning-finished", handle=handle, data={"step": 2})
16finally:
17    nemo_relay.subscribers.flush()
18    nemo_relay.subscribers.deregister("scope-check")

When to Add Marks

Use marks for checkpoints that should appear in the event stream but do not wrap a callback:

  • Request accepted
  • Plan created
  • Memory loaded
  • Routing decision made
  • Final answer assembled
  • Retry scheduled

Use a nested scope instead of a mark when the work has a meaningful start and end boundary, child work, or a duration that matters.

Emit Typed Marks and Metrics

The generic mark API can attach a data_schema describing the opaque data payload and a typed log severity. In Python, pass data_schema= and severity= to scope.event; in Node.js, pass dataSchema and severity as the final event arguments; and in Rust, set data_schema and severity on EmitMarkEventParams. These options let OTLP log export preserve the schema and severity without requiring application-defined metadata keys.

Use the binding’s metric helper for a validated metric group: scope::metric() in Rust, scope.metric() in Python, or metric() in Node.js. See Emit Log and Metric Marks for the metric measurement shape and routing rules.

Validate the Integration

Check that the subscriber prints:

  • One scope start event for agent-run
  • One mark event for planning-started
  • One mark event for planning-finished
  • One scope end event for agent-run

Native subscribers are delivered asynchronously. Await flushSubscribers() and then one event-loop turn before validating printed or captured output.

If marks appear outside the intended trace, pass the active scope handle explicitly or make sure the mark is emitted while the scope is active.

Production Checklist

Use this checklist before running the pattern in production traffic.

  • Keep scope names stable enough for filtering and dashboards.
  • Use marks for business checkpoints, not verbose debug logging.
  • Include only JSON-compatible data, metadata, input, and output payloads.
  • Do not attach sensitive payloads unless a mark or scope event sanitizer removes them before subscriber and exporter delivery.
  • Propagate the active scope stack when work crosses thread or worker boundaries.

Next Steps

Use these links to continue from this workflow into the next related task.