Durable and Scalable Processing#

By default, the REST service processes background analysis within its own process. This mode works without separate workers, but pending in-process work can be lost when the service restarts.

For completed-job processing that persists across API restarts and can scale independently of API traffic, use the durable queue and worker pools. In the current release, this applies to POST /api/v2/process_completed_job; it does not move every API operation into a durable worker.

Start the Workers#

Provision PostgreSQL before enabling durable processing. The API and all workers must use the same database, compatible configuration, and the same Router version. Each process also needs the credentials and storage access required for its role. Installing the package or running aether rest run does not start PostgreSQL or separate worker processes.

Configure the database as described in Installation and Configuration, then configure durable processing on the API and worker hosts before starting the processes:

aether config set rest.task_queue.enabled true
aether config set rest.task_queue.dispatcher.type huey
aether config set rest.task_queue.dispatcher.storage sql
aether config set rest.worker.defaults.recovery_interval_seconds 5

The SQL-backed dispatcher uses tables in the configured PostgreSQL database; it does not require a separate Redis or RabbitMQ service.

Start both worker pools as separate long-running processes, using separate terminals for testing or managed services or containers for deployment:

# Classify completed-job event logs and dispatch analysis.
aether rest worker --pool pcj_classifier
# Run qualification and profiling analysis.
aether rest worker --pool pcj_tools

Once both pools are running, start or restart the API with the updated configuration before accepting completed jobs:

aether rest run

Important

If durable processing is enabled without running workers, completed-job submissions remain queued and do not produce new analysis results. Existing routing decisions can still be served, but migration evidence will not advance until the queued work is processed.

The current release provides the pcj_classifier and pcj_tools pools. It does not provide a separate shadow_evaluation worker pool. Shadow evaluation runs within the completed-job processing flow and can occupy a tools worker while it orchestrates GPU execution.

Scale Deployment Resources#

The API and workers run on CPU infrastructure. GPU Spark execution capacity is separate from the resources assigned to these processes.

Resource

Scaling guidance

API processes or pods

Add replicas behind a load balancer for API traffic and availability. All replicas must share PostgreSQL and compatible configuration. Additional API replicas do not add durable analysis workers.

Classifier workers

Add instances when event-log classification becomes a bottleneck.

Tools workers

Add instances to process more completed jobs concurrently. Size CPU, memory, and temporary disk for the event logs being analyzed, including JVM heap and process overhead.

PostgreSQL

Increase compute, storage, and connection capacity as job history and worker counts grow. Account for connection pools in each process and provision persistent storage and backups.

Event logs, captures, and outputs

Use durable storage accessible to the processes that need it. Local temporary files are not automatically shared across hosts or pods. Size storage and retention for workload volume.

GPU Spark clusters

Provide capacity for the desired number of GPU trials or shadow runs. Adding CPU worker instances does not create additional GPU capacity; account for platform limits and cluster contention.

Start with one classifier and one tools worker, then scale based on backlog, processing time, resource usage, and available GPU execution capacity. Check API health separately from worker health and confirm that accepted jobs complete through the completed-job status API.