Troubleshooting#

Common issues and solutions for the AI-Q blueprint.

Installation Issues#

Issue	Cause	Fix
`ModuleNotFoundError: aiq_agent`	Package not installed in editable mode	`uv pip install -e .`
`nat` command not found	Using system `nat` instead of venv	Use `.venv/bin/nat` or activate the venv
NeMo Agent Toolkit plugins not found	Plugins not installed	`uv pip install -e .` to register entry points
Pre-commit hook failures	Missing pre-commit setup	`pre-commit install && pre-commit run --all-files`
`ormsgpack` attribute error	Version conflict with LangGraph	`uv pip install "ormsgpack>=1.5.0"`

API Key Issues#

Issue	Cause	Fix
`[404] Not found for account`	Invalid or expired NVIDIA API key	Regenerate key at build.nvidia.com
`Gateway timeout (504)`	Model endpoint overloaded or unavailable	Retry, or switch to a different model in config
Tavily search returns empty	Invalid `TAVILY_API_KEY`	Verify key at tavily.com
You.com tools return an unavailable or 401 error	Missing or invalid `YDC_API_KEY`	Create or verify the key using the You.com quickstart and restart AI-Q
Exa search returns empty or 401	Invalid or missing `EXA_API_KEY`	Verify key at exa.ai
Nimble search returns empty or 401	Invalid or missing `NIMBLE_API_KEY`	Verify the key through Nimble
Nimble search returns 403 with “enterprise”	`search_depth: fast` requires an Enterprise plan	Switch to `search_depth: lite` (default) or `deep`, or upgrade your Nimble plan
Serper search fails	Missing `SERPER_API_KEY`	Set key or remove `paper_search_tool` from config

Runtime Issues#

Issue	Cause	Fix
Agent hangs on deep research	LLM timeout or rate limit	Set `verbose: true` in config to see progress; check LLM API availability and rate limits
HTTP 429 or 503 on deep research	Nemotron hosted endpoint availability	Retry after a short delay, reduce concurrency, or follow the self-hosting guidance for consistent throughput
Intermittent shallow-research failure with Nemotron 3.5 Lightning on NVIDIA API Catalog	The hosted serving profile can produce citation-incomplete or malformed final drafts	Use Nemotron Ultra for the shallow role, or use a validated self-hosted Lightning serving profile; see Nemotron 3.5 Lightning on NVIDIA API Catalog
Shallow research returns generic answers	Insufficient tool calls	Increase `max_tool_iterations` (default: 5)
Clarifier keeps asking questions	Too many clarification turns	Reduce `max_turns`, or set `enable_clarifier: false` in the workflow to disable clarification
SSE stream disconnects	Network timeout	Client auto-reconnects using `last_event_id`; refer to Data Flow
Job status stuck on RUNNING	Dask worker crashed	Check Dask logs; the ghost job reaper will eventually mark it FAILURE
OpenShell setup, attestation, readiness, or deletion fails	Gateway, version, policy/config, image, or service-owner mismatch	Follow the canonical OpenShell inspection and troubleshooting guide

Nemotron Hosted Endpoint Availability#

Nemotron 3.5 Lightning (nvidia/nemotron-3.5-lightning-30b-a3b) and Nemotron 3 Ultra (nvidia/nemotron-3-ultra-550b-a55b) are compatible and tested with AIQ, but their NVIDIA-hosted endpoints can have limited availability during high demand. During peak periods you may observe:

Elevated latency or timeouts on LLM inference calls
HTTP 429 (rate-limited) or 503 (service unavailable) responses from the Build API
Degraded agent workflow performance due to upstream model availability

Default Configuration: The default configs use Nemotron 3.5 Lightning for intent classification and shallow research, and Nemotron 3 Ultra for clarification and all deep-research roles. If a hosted endpoint is saturated, retry after a short delay, reduce concurrency, or self-host a downloadable model for consistent throughput.

Recommended Mitigation: Self-Host the Affected Model#

For production and staging deployments that require consistent throughput and low-latency inference, self-host a downloadable NVIDIA NIM rather than relying on shared endpoints. Preview endpoint availability and downloadable NIM availability do not necessarily move in lockstep; verify the current model card before choosing an image.

Self-host Nemotron 3.5 Lightning 30B A3B for the default intent and shallow-research roles
Self-host Nemotron 3 Ultra 550B A55B for the default clarification and deep-research roles

Once your self-hosted endpoint is running, update the corresponding base_url in your config to point at it. AIQ’s configuration validator currently requires NVIDIA_API_KEY for every _type: nim profile, even when a local NIM does not enforce client authentication. Set a non-secret placeholder for the local deployment before starting AIQ:

export NVIDIA_API_KEY=local-nim

Then reference that variable in the local profile:

llms:
  local_ultra_llm:
    _type: nim
    # Use the identifier returned by the local NIM's /v1/models endpoint.
    model_name: nvidia/nemotron-3-ultra-550b-a55b
    base_url: "https://<your-ultra-endpoint>/v1"
    api_key: ${NVIDIA_API_KEY}
    temperature: 0.2
    top_p: 0.7
    max_tokens: 16384
    num_retries: 5
    chat_template_kwargs:
      enable_thinking: false

Nemotron 3.5 Lightning on NVIDIA API Catalog#

The default profiles retain Nemotron 3.5 Lightning for intent classification and shallow research. When the shallow role uses Lightning through the NVIDIA API Catalog endpoint (integrate.api.nvidia.com), the hosted serving profile can intermittently return citation-incomplete or malformed final drafts. AI-Q verifies the draft against the captured source registry and fails closed instead of publishing an unsupported answer, so an affected request ends with a failed workflow outcome even when its search completed successfully.

This behavior depends on the serving profile, not only the model weights. It did not reproduce at the same rate in validation with the tested self-hosted NVFP4 vLLM profile. For a deployment that prioritizes shallow-answer reliability, use one of these configurations:

Assign Nemotron Ultra to shallow_research_agent.llm, while keeping Lightning for intent classification.
Serve Lightning through a self-hosted profile that you validate end to end with AI-Q’s citation and tool-calling workflow.

The Brev getting-started launchable uses the first option. This keeps the launchable reliable without changing the model assignment in the general-purpose shipped profiles.

Knowledge Layer Issues#

Issue	Cause	Fix
`Unknown backend`	Adapter module not imported	Ensure backend package is installed: `uv pip install -e "sources/knowledge_layer[llamaindex]"`
Empty retrieval results	Ingestion and retrieval resolved different collections	Verify the upload-path collection and active `conversation-id`; without session context, verify the configured `collection_name` fallback
Foundational RAG connection refused	RAG Blueprint not running	Start the RAG Blueprint server; verify `rag_url` and `ingest_url`
`milvus-lite` required	Missing dependency	`uv pip install "pymilvus[milvus_lite]"`

Docker / Deployment Issues#

Issue	Cause	Fix
Container fails to start	Missing environment variables	Check `deploy/.env` has all required keys
Port already in use	Another service on port 3000/8000	Set `PORT=8100` or `FRONTEND_PORT=3100` in `.env`
UI shows “Backend unavailable”	Backend not healthy	`curl http://localhost:8000/health`; check backend container logs

VM / Remote Development#

If you are running the AI-Q blueprint on a remote VM (cloud instance, WSL, SSH server) and accessing it from your local browser, localhost:3000 and localhost:8000 will not resolve because the services are listening on the VM — not your local machine.

SSH Port Forwarding#

Forward the required ports through your SSH connection:

# Forward both the frontend and backend ports
ssh -L 3000:localhost:3000 -L 8000:localhost:8000 user@your-vm-host

Then open http://localhost:3000 on your local machine as usual.

To forward ports to an already-active SSH session, you can also use ~C (SSH escape sequence) to open the SSH command line and type the following on a single line (press Enter at the end):

-L 3000:localhost:3000 -L 8000:localhost:8000

VS Code Remote SSH#

If you are using VS Code Remote-SSH, ports are typically forwarded automatically when the server starts listening. If not, open the Ports panel (Ctrl+Shift+P → “Ports: Focus on Ports View”) and add ports 3000 and 8000 manually.

Common Symptoms#

Symptom	Cause	Fix
“This site can’t be reached” on `localhost:3000` or `localhost:8000`	Ports not forwarded from VM to local machine	Use SSH port forwarding (see above)
Connection refused after forwarding	Service not running on the VM	SSH into the VM and verify with `curl http://localhost:8000/health`
Port forwarding conflicts	Local port already in use	Use alternate local ports: `ssh -L 3001:localhost:3000 -L 8001:localhost:8000 user@vm`

Note

Docker Compose deployments on the VM handle container-to-host port mapping automatically. The SSH forwarding described here is for making the VM’s ports accessible on your local machine.

Debugging Tips#

Enable Verbose Logging#

# In your config YAML
workflow:
  _type: chat_deepresearcher_agent
  verbose: true

Or through CLI: ./scripts/start_cli.sh --verbose

Phoenix Tracing#

For full setup instructions covering Phoenix, LangSmith, and other tracing backends, see Observability.

Start a Phoenix server and enable tracing in config:

general:
  telemetry:
    tracing:
      phoenix:
        _type: phoenix
        endpoint: http://localhost:6006/v1/traces
        project: dev

Then open http://localhost:6006 to inspect traces, token usage, and latency.

Check Registered Components#

# List registered NeMo Agent Toolkit plugins
.venv/bin/nat info components