Sweeper Tutorial

1. Define the Backend Search

Choose a model, hardware system, deployment modes, backends, and GPU budget:

1
search_space:
2
  model_name: meta-llama/Meta-Llama-3.1-8B
3
  hardware_sku: gb200
4
  deployment_mode: [agg]
5
  backend: [trtllm]
6
  gpu_budget: 8

Sweeper enumerates legal parallel configurations, removes unsupported runner topologies, and adds the active engine-role knobs to the optimizer study.

2. Define One Workload and Goal

1
workload:
2
  isl: 1024
3
  osl: 128
4
  request_rate: 4
5
  num_request_ratio: 10
6
7
goal:
8
  target: throughput

Every candidate is evaluated against this workload. See Traffic for trace and closed-loop alternatives, and Optimization Goals for SLA and Pareto modes.

3. Control the Sweep

1
sweep:
2
  max_rounds: 2
3
  candidates_per_round: 4
4
  parallel_evals: 2
5
  max_eval_seconds: 300

Each round is a barrier: the sampler asks for unique candidates, replay evaluates them, and then the scores are reported back together. parallel_evals controls replay worker fan-out.

4. Add Optional Feature Search

An installed or injected provider owns the schema below its adapter name:

1
adapters:
2
  example.policy:
3
    search_space:
4
      mode: [balanced, latency]

The provider receives the complete search_space mapping. It does not receive a preselected concrete feature configuration.

5. Run and Inspect

1
sweeper = Sweeper(
2
    runner_factory=my_runner_factory,
3
    providers={"example.policy": my_provider},
4
)
5
candidates = sweeper.run(config)
6
7
for candidate in candidates[:5]:
8
    print(candidate.score, candidate.used_gpus, candidate.config)

The same Sweeper instance can run multiple configurations. Studies, caches, and process pools are new for every call.