Operations and Troubleshooting
Use the service manager to start and stop the daemon. The harness starts MCP,
and the daemon broker controls worker startup and shutdown. The
nemo-relay gateway stop command stops a personal gateway. Stop a managed daemon
through its service manager.
Verify Worker-Backed Operation
Run these checks before rollout and after an upgrade. Use a test session and a non-sensitive prompt. Keep the harness open throughout the request checks.
Validate the Bundle
Run nemo-relay doctor --managed-bundle <installed-bundle-path> --managed-bundle-sha256 <trusted-digest>
in the user’s prepared environment. Expect bundle and environment validation
to succeed. The details report the route credential source, env or file.
This does not prove live daemon or worker readiness.
Confirm Worker Readiness
Start the managed harness. Check its MCP status and the worker’s configured log
destinations, as described in Find Logs. With info-level JSONL
logging enabled, expect configuration_resolved with mode = "managed_worker".
Then look for worker_ready with the message Daemon worker is ready.
Confirm the MCP handshake in the harness status view. For a reused worker,
find its earlier ready event and confirm the worker still exists.
Query the daemon’s aggregate worker status while the harness remains open:
Confirm that the expected worker UUID appears with state = "ready" and
control_available = true. Use
/_nemo_relay/v1/workers/<instance_uuid>/status to inspect that worker alone.
Check Worker Reachability
Confirm that the worker’s endpoint is on the client computer and reachable from the daemon. Use the listener and network checks in your runbook. A process listing alone is insufficient.
Exercise Model Requests and Hooks
Send a model request through the harness. Confirm streaming output arrives before completion and that the selected provider URL is the daemon origin. Trigger a harmless tool operation to exercise a managed hook.
Check Plugin Results
Check an observable result from your configured worker plugin: for example, new ATOF records in the approved collector for this session, or the expected outcome of a test policy. Confirm both model and hook activity where supported.
Verify Sharing and Shutdown
Open a second harness for the same user-machine identity. Confirm it reuses the worker. Close both sessions, allow up to two minutes for accepted requests to drain, and confirm the worker exits. If a process disappeared without unregistering, allow the 30-second reconnect grace period first.
A successful provider response or daemon_mcp_ready event does not prove worker
execution. Pass-through can also produce both. Do not accept a deployment until
the worker and expected plugin behavior have been observed.
An empty worker list also does not prove a failure. Routes that use passthrough
or are starting a worker have no assigned worker UUID. They do not appear in
the list. Compare /healthz, worker status, lifecycle events, and a plugin
result to confirm that traffic used a worker.
For a remote deployment, also test a planned daemon restart and a VPN reconnect. Verify recovery or start a fresh harness if the old one cannot recover its control session. Check the documented streaming limitations before selecting middleware that rewrites responses.
Find Logs
Start with the process’s logging settings. File sinks send logs to configured
files. Relay also writes to stderr when stderr_enabled is true. Logging flags,
--log-config-path, and logging environment variables can override settings
from discovered configuration files. The service manager or harness decides
where to capture stderr.
Use this table to find the settings that control each component’s logs:
By default, worker stderr goes to the launcher’s or harness’s capture. Relay does not create worker log files under daemon state.
The supplied service runbooks use these default locations to capture daemon stderr:
Check the installed service or task settings if your deployment changes these locations. Stderr output must be enabled for these captures to contain Relay logs.
Set level = "info" and stderr_format = "jsonl" in [daemon.logging] to
record daemon lifecycle messages. Refer to
Configuration for details.
The primary daemon does not inherit personal [logging] settings.
Managed MCP and hook commands do not look for logging files automatically.
To record their info messages, supply a log configuration file or pass
NEMO_RELAY_LOG=info and NEMO_RELAY_LOG_STDERR_FORMAT=jsonl through the harness
or dispatcher. Otherwise, use the harness MCP status view. The optional
daemon_mcp_ready log can be absent. Do not edit managed bundle files to enable
logging. Do not send debug output to MCP stdout.
Record the time, daemon origin, component, event, and error kind when diagnosing an incident. Do not include route tokens, provider keys, private identity files, or activation grants in support reports.
Relay processes emit these lifecycle events. Normal events use info-level logging. Failure events use warning or error levels.
Use fingerprint to match an MCP session to its worker route. Use worker_id
to follow one worker generation. A route_mode = "pass_through" field reports
a route without worker forwarding. With --require-worker, that route returns
503. Share session and worker IDs only within your approved support process.
When a bound route passes a request through, the daemon logs
public_request_pass_through at info level with route_mode = "pass_through".
reason = "worker_unavailable" means the worker is not ready or its control
connection is down, including during the recovery grace period.
reason = "configured_pass_through" identifies --pass-through deployments.
The route field identifies hook, openai, or anthropic. The daemon logs
these events for every request, including repeated permission hooks.
Both worker_launch_failed and worker_activation_failed include a
failure_reason. This is a short, fixed code that names the step that failed,
such as starting the worker, sending its startup permission, or waiting for it to
be ready. It does not include activation data or the source error text. Read
the MCP launcher’s configured logs for the worker_launch_failed message.
Its error text is separate from failure_reason and includes the source error.
Windows spawn error text identifies the failed operation and OS error code.
Worker logs may not exist if process creation failed.
worker_communication_failed includes a reason that identifies a transport
error, invalid destination or response, response-body error, or explicit worker
route rejection. worker_request_failed with reason = "response_head_timeout"
reports a timed-out request without disconnecting the worker or changing its
route. Check worker and provider logs to identify the delay.
Control disconnect events include a reason such as acknowledgement_timeout,
pong_timeout, peer_closed, or worker_communication_failed. A staged worker
cancelled when its launcher exits reports activation_cancelled. This event does
not mean that the startup deadline expired. Provider failure events are rate-limited and
report the number of intervening failures in suppressed_since_last_emit.
Diagnose Common Failures
Use this table to choose the next check for a failure:
Older Python plugin environments without .nemo-relay-environment.key need the
original installer’s bootstrap key to verify their attestations. If that key
is missing or unreadable, reprovision the environment from the trusted plugin
source. Newly provisioned environments store the key with the environment.
Worker startup has no deadline or retry limit. A slow worker keeps its current launch grant. Check the worker’s configured logs to find out why startup is slow.
Use these daemon events to track startup and retries:
worker_startup_pendingreportselapsed_msevery 60 seconds during startup.worker_addedrecordsstartup_elapsed_mswhen the worker joins the route.worker_activation_failedreports the retry delay inretry_after_ms.worker_relaunch_requestedrecords the attempt count.
If startup fails, retry delays increase from one second to two, four, and so on. They stop increasing at 60 seconds. Authentication failures and control failures that cannot recover still end MCP. Disconnected MCP clients keep their route references during recovery, but they cannot receive launch grants.
By default, authenticated requests use passthrough during startup and recovery.
Use nemo-relay daemon --require-worker when every request needs worker
guardrails, plugins, or observability. This strict mode returns HTTP 503 during
startup, recovery, and retry delays. Configure the calling application to handle
those responses. The --pass-through flag disables workers and conflicts with
--require-worker.
A request or stream keeps its original destination when traffic switches to the worker. Relay never resubmits it to a different destination.
Keep Identity State
The state base is XDG_CONFIG_HOME when set. Otherwise Relay uses HOME/.config
if HOME is set, then USERPROFILE/.config if USERPROFILE is set. This order
also applies on Windows: HOME wins over USERPROFILE. Relay stores daemon
state below nemo-relay/daemon in that base directory.
Before backup or recovery, check these variables in the environment of the
account and process that owns the state. An interactive shell can differ from a
service or managed harness. The runbooks give system services an explicit,
persistent XDG_CONFIG_HOME. Client processes can use a different base.
daemon-identity.pk8 is the daemon signing key. machine-identity.pk8 identifies
the client machine-user. Trust pins are stored under pins and are scoped to the
normalized daemon origin. These keys are separate from the server’s HTTPS key
and certificate.
Back up private state through an access-controlled backup system and preserve ownership when restoring it. Do not run two machines with a cloned client identity. Do not delete trust pins just to silence a mismatch: first confirm whether the daemon was intentionally replaced or the connection is unexpected. An intended signing-identity replacement needs a coordinated, verified client trust update. A normal TLS certificate renewal does not require replacing the Relay signing key.
Route credential bindings are in memory. A daemon restart requires MCP re-registration, even when signing identity is preserved. There is no documented per-user credential-revocation CLI. For offboarding, remove the user’s network access and managed launch access and end their sessions. Do not assume deleting a local token file revokes a binding from the running broker. A new token for the same machine identity joins its existing route, up to the per-identity token limit (4 by default). Earlier tokens stay bound until the daemon restarts, an idle one is released at that limit, or the identity’s whole route is evicted from a full route table. Until MCP re-registers after a restart, that user’s requests use pass-through.
Upgrade and Roll Back
Follow these steps to upgrade the daemon and clients:
Prepare the Upgrade
Schedule a maintenance window. Save the approved binary version, service definition, configuration, and separately trusted bundle digest. Verify that the replacement daemon and client binaries are compatible before rollout.
Drain Sessions
Ask users to close affected harnesses. Let accepted requests drain and wait for workers to exit. A service-manager stop can interrupt active streams.
Stop Processes
Stop the daemon with the matching service manager. On clients, ensure no MCP or worker process still uses the binary, especially on Windows.
For rollback, stop affected processes and restore the previous verified binary and compatible service/configuration files. Keep the same persistent identities and validate the same bundle. Restore a compatible set of daemon and client binaries together.
Use a separately named artifact for an incompatible managed-settings change. When a plugin configuration changes, restart affected harness sessions so a fresh worker loads the new system configuration.
Remove a Deployment
Use your platform’s task, launchd, or systemd removal steps. Remove the managed Relay plugin registration and provider settings through the deployment mechanism that installed them. Preserve unrelated harness settings and authentication. Remove firewall rules only when they no longer serve a remaining deployment.
Keep identity state and the old bundle for the rollback period. Delete private state and bootstrap tokens only as part of the intended final retirement. Stopping the daemon does not remove client-side files or undo harness settings.