Observing a Workflow with Catalyst#

This guide provides a step-by-step process to enable observability in a NeMo Agent toolkit workflow using Catalyst for tracing. By the end of this guide, you will have:

  • Configured telemetry in your workflow.

  • Ability to view traces in the Catalyst platform.

Step 1: Sign up for Catalyst#

Step 2: Create a Project#

After logging in, create a new project.

  • Project Name: Choose any name.

  • Use Case: Agentic Application

Step 3: Generate API Credentials#

Go to your profile settings to generate your:

  • Access Key

  • Secret Key

Step 4: Configure Your Environment#

Set the following environment variables in your terminal:

export CATALYST_ACCESS_KEY=<your_access_key>
export CATALYST_SECRET_KEY=<your_secret_key>
export CATALYST_ENDPOINT=https://catalyst.raga.ai/api

Step 5: Install the RagAI Subpackage#

uv pip install -e '.[ragaai]'

Step 6: Modify Workflow Configuration#

Update your workflow configuration file to include the telemetry settings.

Example configuration:

general:
  telemetry:
    tracing:
      catalyst:
        _type: catalyst
        project: catalyst-demo
        dataset: catalyst-dataset
        tracer_type: my-tracer-type
        endpoint: ${CATALYST_ENDPOINT}
        access_key: ${CATALYST_ACCESS_KEY}
        secret_key: ${CATALYST_SECRET_KEY}

Step 7: Run Your Workflow#

From the root directory of the NeMo Agent toolkit library, install dependencies and run the pre-configured simple_calculator_observability example.

Example:

# Install the workflow and plugins
uv pip install -e examples/observability/simple_calculator_observability/

# Run the workflow with Catalyst telemetry settings
# Note, you may have to update configuration settings based on your Catalyst account
nat run --config_file examples/observability/simple_calculator_observability/configs/config-catalyst.yml --input "What is 1*2?"

As the workflow runs, telemetry data will start showing up in Catalyst.

Step 8: View Traces Data in Catalyst#

  • Open your browser and navigate to https://catalyst.raga.ai/projects.

  • Locate your workflow traces under your configured project name and dataset.

  • Inspect function execution details, latency, total tokens, request timelines and other info under Info and Attributes tabs of an individual trace.

Catalyst Trace View

Debugging#

If you encounter issues while downloading the Catalyst package, try uninstalling and installing:

uv pip uninstall ragaai-catalyst

uv pip install ragaai-catalyst