Observe Workflows#

The NeMo Agent toolkit uses a flexible, plugin-based observability system that provides comprehensive support for configuring logging, tracing, and metrics for workflows. Users can configure multiple telemetry exporters simultaneously from the available options or create custom integrations. The observability system:

  • Uses an event-driven architecture with IntermediateStepManager publishing workflow events to a reactive stream

  • Supports multiple concurrent telemetry exporters processing events asynchronously

  • Provides built-in exporters for popular observability platforms (Phoenix, Langfuse, Weave, etc.)

  • Enables custom telemetry exporter development for any observability service

These features enable developers to test their workflows locally and integrate observability seamlessly with their preferred monitoring stack.

Compatibility with Previous Versions#

As of v1.2, the span exporter exports attributes names prefixed with nat by default. In prior releases the attribute names were prefixed with aiq, to retain compatibility the NAT_SPAN_PREFIX environment variable can be set to aiq:

export NAT_SPAN_PREFIX=aiq

Installation#

The core observability features (console and file logging) are included by default. For advanced telemetry features like OpenTelemetry and Phoenix tracing, you need to install the optional telemetry extras:

# Install all optional telemetry extras
uv pip install -e '.[telemetry]'

# Install specific telemetry extras
uv pip install -e '.[opentelemetry]'
uv pip install -e '.[phoenix]'
uv pip install -e '.[weave]'
uv pip install -e '.[ragaai]'

Configurable Components#

The flexible observability system is configured using the general.telemetry section in the workflow configuration file. This section contains two subsections: logging and tracing, and each subsection can contain multiple telemetry exporters running simultaneously.

For a complete list of logging and tracing plugins and corresponding configuration settings use the following CLI commands.

# For all registered logging plugins
nat info components -t logging

# For all registered tracing plugins
nat info components -t tracing

Illustrated below is a sample configuration file demonstrating multiple exporters configured to run concurrently.

general:
  telemetry:
    logging:
      console:
        _type: console
        level: WARN
      file:
        _type: file
        path: ./.tmp/workflow.log
        level: DEBUG
    tracing:
      # Multiple exporters can run simultaneously
      phoenix:
        _type: phoenix
        # ... configuration fields
      weave:
        _type: weave
        # ... configuration fields
      file_backup:
        _type: file
        # ... configuration fields

Logging Configuration#

The logging section contains one or more logging providers. Each provider has a _type and optional configuration fields. The following logging providers are supported by default:

  • console: Writes logs to the console.

  • file: Writes logs to a file.

Tracing Configuration#

The tracing section contains one or more tracing providers. Each provider has a _type and optional configuration fields. The observability system supports multiple concurrent exporters.

Available Tracing Exporters#

Each exporter has its own detailed configuration guide with complete setup instructions and examples:

For complete configuration examples and setup instructions, refer to the individual guides linked above or check the examples/observability/ directory.

NeMo Agent Toolkit Observability Components#

The NeMo Agent toolkit observability system uses a generic, plugin-based architecture built on the Subject-Observer pattern. The system consists of several key components working together to provide comprehensive workflow monitoring:

Event Stream Architecture#

  • IntermediateStepManager: Publishes workflow events (IntermediateStep objects) to a reactive event stream, tracking function execution boundaries, LLM calls, tool usage, and intermediate operations.

  • Event Stream: A reactive stream that broadcasts IntermediateStep events to all subscribed telemetry exporters, enabling real-time observability.

  • Asynchronous Processing: All telemetry exporters process events asynchronously in background tasks, keeping observability “off the hot path” for optimal performance.

Telemetry Exporter Types#

The system supports multiple exporter types, each optimized for different use cases:

  • Raw Exporters: Process IntermediateStep events directly for simple logging, file output, or custom event processing.

  • Span Exporters: Convert events into spans with lifecycle management, ideal for distributed tracing and span-based observability services.

  • OpenTelemetry Exporters: Specialized exporters for OTLP-compatible services with pre-built integrations for popular observability platforms.

  • Advanced Custom Exporters: Support complex business logic, stateful processing, and enterprise reliability patterns with circuit breakers and dead letter queues.

Processing Pipeline System#

Each exporter can optionally include a processing pipeline that transforms, filters, batches, or aggregates data before export:

  • Processors: Modular components for data transformation, filtering, batching, and format conversion.

  • Pipeline Composition: Chain multiple processors together for complex data processing workflows.

  • Type Safety: Generic type system ensures compile-time safety for data transformations through the pipeline.

Integration Components#

  • nat.profiler.decorators: Decorators that wrap workflow and LLM framework context managers to inject usage-collection callbacks.

  • callbacks: Callback handlers that track usage statistics (tokens, time, inputs/outputs) and push them to the event stream. Supports LangChain, LLama Index, CrewAI, and Semantic Kernel frameworks.

Registering a New Telemetry Provider as a Plugin#

For complete information about developing and integrating custom telemetry exporters, including detailed examples, best practices, and advanced configuration options, see Adding Telemetry Exporters.