Durable and Scalable Processing#
By default, the REST service processes background analysis within its own process. This mode works without separate workers, but pending in-process work can be lost when the service restarts.
For completed-job processing that persists across API restarts and can scale
independently of API traffic, use the durable queue and worker pools. In
the current release, this applies to POST /api/v2/process_completed_job; it does not
move every API operation into a durable worker.
Start the Workers#
Provision PostgreSQL before enabling durable processing. The API and all
workers must use the same database, compatible configuration, and the same
Router version. Each process also needs the credentials and storage access
required for its role. Installing the package or running aether rest run
does not start PostgreSQL or separate worker processes.
Configure the database as described in Installation and Configuration, then configure durable processing on the API and worker hosts before starting the processes:
aether config set rest.task_queue.enabled true
aether config set rest.task_queue.dispatcher.type huey
aether config set rest.task_queue.dispatcher.storage sql
aether config set rest.worker.defaults.recovery_interval_seconds 5
The SQL-backed dispatcher uses tables in the configured PostgreSQL database; it does not require a separate Redis or RabbitMQ service.
Start both worker pools as separate long-running processes, using separate terminals for testing or managed services or containers for deployment:
# Classify completed-job event logs and dispatch analysis.
aether rest worker --pool pcj_classifier
# Run qualification and profiling analysis.
aether rest worker --pool pcj_tools
Once both pools are running, start or restart the API with the updated configuration before accepting completed jobs:
aether rest run
Important
If durable processing is enabled without running workers, completed-job submissions remain queued and do not produce new analysis results. Existing routing decisions can still be served, but migration evidence will not advance until the queued work is processed.
The current release provides the pcj_classifier and pcj_tools pools.
It does not provide a separate shadow_evaluation worker pool. Shadow
evaluation runs within the completed-job processing flow and can occupy a
tools worker while it orchestrates GPU execution.
Scale Deployment Resources#
The API and workers run on CPU infrastructure. GPU Spark execution capacity is separate from the resources assigned to these processes.
Resource |
Scaling guidance |
|---|---|
API processes or pods |
Add replicas behind a load balancer for API traffic and availability. All replicas must share PostgreSQL and compatible configuration. Additional API replicas do not add durable analysis workers. |
Classifier workers |
Add instances when event-log classification becomes a bottleneck. |
Tools workers |
Add instances to process more completed jobs concurrently. Size CPU, memory, and temporary disk for the event logs being analyzed, including JVM heap and process overhead. |
PostgreSQL |
Increase compute, storage, and connection capacity as job history and worker counts grow. Account for connection pools in each process and provision persistent storage and backups. |
Event logs, captures, and outputs |
Use durable storage accessible to the processes that need it. Local temporary files are not automatically shared across hosts or pods. Size storage and retention for workload volume. |
GPU Spark clusters |
Provide capacity for the desired number of GPU trials or shadow runs. Adding CPU worker instances does not create additional GPU capacity; account for platform limits and cluster contention. |
Start with one classifier and one tools worker, then scale based on backlog, processing time, resource usage, and available GPU execution capacity. Check API health separately from worker health and confirm that accepted jobs complete through the completed-job status API.