ThunderAgent Scheduler Reference

Control-loop flags, constraints, capacity behavior, and scheduler log records

View as Markdown

dynamo.thunderagent_router is experimental. The command-line interface and lifecycle contract can change.

The ThunderAgent scheduler uses the program working set as a fraction of each worker’s retention budget. With SGLang HiCache enabled, the budget is the sum of published GPU KV capacity and host HiCache capacity. Mooncake capacity is excluded because it is conditional content-addressed storage rather than guaranteed per-program retention.

Control-Loop Configuration

FlagEnvironment variableDefaultDescription
--pause-thresholdDYN_THUNDERAGENT_PAUSE_THRESHOLD0.95Working-set fraction of the retention budget that starts a pause cycle.
--soft-demote-thresholdDYN_THUNDERAGENT_SOFT_DEMOTE_THRESHOLD0.80Start of the soft-demote band below the pause threshold.
--pause-targetDYN_THUNDERAGENT_PAUSE_TARGET0.80Setpoint that a pause cycle drives utilization toward.
--resume-hysteresisDYN_THUNDERAGENT_RESUME_HYSTERESIS0.10Headroom below pause-threshold required before resume.
--resume-priority-boostDYN_THUNDERAGENT_RESUME_PRIORITY_BOOST1.0Transient priority seconds added to a resumed request.
--resume-timeout-secondsDYN_THUNDERAGENT_RESUME_TIMEOUT_SECONDS1800.0Forced-resume cap that bounds starvation.
--scheduler-interval-secondsDYN_THUNDERAGENT_SCHEDULER_INTERVAL_SECONDS5.0Scheduler tick period.
--soft-demote-priority-jumpDYN_THUNDERAGENT_SOFT_DEMOTE_PRIORITY_JUMP-2.0Priority seconds applied to soft-demoted programs.
--acting-token-weightDYN_THUNDERAGENT_ACTING_TOKEN_WEIGHT1.0Multiplier on token_total for acting programs in the pause-side working set.
--acting-decay-tau-secondsDYN_THUNDERAGENT_ACTING_DECAY_TAU_SECONDS1.0Exponential-decay time constant for acting tokens in the resume-side working set.

All KvRouter flags accepted by dynamo.router, including --router-temperature, --use-kv-events, and --router-track-output-blocks, are also accepted and forwarded.

Constraints

The service rejects configurations that violate any of these relationships:

0 <= pause_threshold <= 1
0 <= pause_target <= pause_threshold
0 <= resume_hysteresis <= pause_threshold
0 <= soft_demote_threshold <= pause_threshold
acting_token_weight > 0
scheduler_interval_seconds > 0
resume_timeout_seconds > 0

The control loop has three bands:

  • At or above pause-threshold, pause acting programs until utilization reaches pause-target.
  • From soft-demote-threshold up to pause-threshold, lower program priority without pausing.
  • Resume only after utilization falls at least resume-hysteresis below pause-threshold.

Session Inputs

Programs are keyed by the normalized session_id. Use the canonical headers defined in Session IDs:

  • X-Dynamo-Session-ID
  • X-Dynamo-Parent-Session-ID
  • X-Dynamo-Session-Final

Requests without session identity bypass program admission and pause/resume.

Scheduler Logs

The scheduler emits pause-side and resume-side summaries at INFO level.

Pause-side summary
scheduler.tick worker=<id> paused=<N> marked=<M> util=<X> -> <Y>
  • paused: acting programs paused during the tick
  • marked: reasoning programs marked to pause at the next tool boundary
  • util: worker utilization before and after the pause cycle
Resume-side summary
scheduler.tick resumed=<N> still_paused=<M>
  • resumed: programs resumed during the tick
  • still_paused: programs remaining in the paused table

Lower the log level for dynamo.thunderagent_router to DEBUG for per-program records:

Paused program <program_id> (tokens=<n>)
Resumed program <program_id> -> worker=<id> (tokens=<n>)

For request-level tracing, enable DYN_REQUEST_TRACE=1 on the frontend. Explicit harness tool spans additionally require DYN_REQUEST_TRACE_TOOL_EVENTS_ZMQ_ENDPOINT and a configured publisher. See Request Trace Reference.