> For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/aiperf/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/aiperf/_mcp/server. # Memory Estimator AIPerf ships a static memory estimator that predicts the peak and steady-state RSS of every pod in a Kubernetes deployment from the benchmark configuration. It is an **advisory** report: `aiperf kube generate`, `aiperf kube profile`, and the operator preflight all run it and surface its output, but none of them rewrite the pod's `resources` block from it. Actual requests/limits come from the `AIPERF_K8S_*` resource settings in `src/aiperf/kubernetes/environment.py`. If your `records-manager` OOMs at 500k concurrency, if workers get killed during ramp, or if you need to justify a resource bump — read this page first. --- ## Purpose The estimator answers three related questions: 1. **Will this workload fit in the current memory limits?** Each `PodEstimate` carries both the projected peak RSS and the configured K8s limit; the `headroom_pct` / `at_risk` properties flag OOM risk before a job is submitted. 2. **What limits should I set?** `recommended_request_mib` (steady-state × 1.2) and `recommended_limit_mib` (peak × 1.3) come straight off the estimate. 3. **Where is the memory actually going?** Per-component breakdowns identify whether `RecordsManager` growable arrays, `RecordProcessor` tokenizer caches, or in-flight `Worker` records dominate — so tuning targets the right knob. The model is static: formulas are derived from code inspection, and the constants come from two different kinds of calibration (see the provenance notes on each constant in `src/aiperf/kubernetes/_memory_estimator/constants.py`). No runtime profiling is required at estimate time. - **Aggregate MiB baselines** — `_PYTHON_SUBPROCESS_BASE_MIB`, `_SERVICE_BASE_MIB`, `_TOKENIZER_CACHE_MIB`, and the margin multipliers were measured against real-cluster `container_memory_working_set_bytes` during the 2026-04-30 ISL/OSL sweep. Re-derive them only from a new cluster sweep. - **Per-object byte constants** — `_REQUEST_RECORD_BASE_BYTES`, `_TURN_BASE_BYTES`, and the SSE / `TextResponse` terms are measured in-process against the real model classes, as *amortized marginal* heap bytes per instance via `tracemalloc` snapshot diffs over 1000–3000 live instances after a warmup pass. Amortizing is required: the one-off cost of shared field-name strings and Pydantic validator objects is paid by the first instance and must not be charged to the marginal one. `sys.getsizeof` is not usable — it sees only the top-level object and misses `__dict__` / `__pydantic_extra__` and every referenced string. `TestPerRequestBytesAgainstMeasuredHeap` in `tests/unit/kubernetes/test_memory_estimator.py` re-runs that measurement on every test run and fails if `_per_request_bytes` drifts off the conservative side of it, so the per-object constants are reproducible without any external script. **Consumers of estimator output:** - `aiperf kube generate` — prints the full estimate to **stderr** (so stdout stays a clean `kubectl apply -f -` stream) after emitting the manifests. - `aiperf kube profile` — prints the full estimate and warnings to **stderr** before submitting the job. - Operator preflight (`src/aiperf/operator/preflight/_resources.py`) — runs the estimator as the `Memory Estimation` check. If `estimate.warnings` is non-empty the check returns `WARN` with `estimate.recommendations` as hints; it does **not** fail or reject the job. --- ## Inputs All inputs are captured in `MemoryEstimationParams` (`src/aiperf/kubernetes/_memory_estimator/params.py`). They fall into four groups: ### Topology | Field | Meaning | |---|---| | `total_workers` | Total worker processes across all pods. | | `workers_per_pod` | Worker processes per worker pod. | | `num_worker_pods` | Worker pod replica count (`ceil(total_workers / workers_per_pod)`). | | `record_processors_per_pod` | Record processors per worker pod. `benchmark.runtime.record_processors_per_pod` when set, otherwise `max(1, workers_per_pod // RECORD_PROCESSOR_SCALE_FACTOR)`. | `_derive_topology` reads the same config fields, in the same precedence order, that `spec_converter.apply_worker_config` uses to lay out the JobSet, so the estimate always describes the pods that are actually created. The operator calls `apply_worker_config` before preflight, which normalizes `workers_per_pod` and `record_processors_per_pod` onto the config first — so the estimate also picks up the cluster-wide `runtime.record_processors` total (divided across pods) and the single-pod collapse that a worker count not divisible by `workers_per_pod` triggers. The `aiperf kube generate` and `aiperf kube profile --dry-run` banners print their estimate from the *pre-normalization* config. An explicit `recordProcessorsPerPod` or `workersPerPod` is honored there, but a bare `runtime.record_processors` total, and the non-divisible single-pod collapse, are not yet resolved at that point and so are not reflected in the printed numbers. ### Load profile | Field | Meaning | |---|---| | `max_concurrency` | Peak in-flight request concurrency across all workers. | | `total_requests` | Total requests for the entire benchmark (sum across phases). | | `total_benchmark_duration_s` | Total benchmark wall-clock seconds. | ### Dataset shape | Field | Meaning | |---|---| | `dataset_count` | Number of conversations pre-synthesized. | | `avg_isl_tokens` | Weighted-mean input sequence length in tokens. | | `avg_osl_tokens` | Weighted-mean output sequence length in tokens. | | `max_turns` | Maximum conversation turns (1 for single-turn). | | `streaming` | SSE vs. buffered-text responses (changes per-chunk accounting). | | `list_metric_backend` | `ragged` keeps every inter-chunk-latency value; `tdigest` uses one bounded sketch. Derived from `AIPERF_METRICS_LIST_BACKEND`. | ### Observability | Field | Meaning | |---|---| | `num_gpus`, `gpu_sample_interval_s`, `num_gpu_metrics` | DCGM sampling shape. | | `num_server_metrics_endpoints`, `server_metrics_scrape_interval_s` | Prometheus scrape shape. | | `est_unique_metric_series`, `est_histogram_metrics`, `est_histogram_buckets` | Per-endpoint series cardinality. | | `num_models`, `num_standard_metrics`, `export_http_trace` | Per-RP tokenizer count, per-record metrics, trace export flag. | `MemoryEstimationParams.from_config(config, total_workers, workers_per_pod, connections_per_worker)` is the normal entry point: it derives all of the above from an `AIPerfConfig` plus three deployment parameters. --- ## Per-component model Every per-process estimate returns a `ComponentEstimate` (see `src/aiperf/kubernetes/_memory_estimator/estimates.py`) with four numeric fields — `base_mib`, `variable_mib`, `peak_mib`, and a derived `steady_state_mib = base_mib + variable_mib`. Two universal baselines ride along on every process: - `_PYTHON_SUBPROCESS_BASE_MIB = 150` — interpreter + core libs + GC + every module an AIPerf service loads (numpy, pandas, msgspec, pydantic, orjson, aiohttp, ZMQ, asyncio). Calibrated from a real-cluster ISL/OSL sweep (2026-04-30) to a ~150 MiB common baseline per container. - `_PYTHON_CHILD_SUBPROCESS_BASE_MIB = 150` — Worker/RP subprocesses are modeled the same as the parent. Copy-on-write does not materially shrink the measured working set (`container_memory_working_set_bytes` is per-container, and each process's heap diverges quickly once it allocates per-task state). Each service adds a per-service overhead from `_SERVICE_BASE_MIB` (e.g. `records_manager: 40`, `dataset_manager: 30`, `worker: 12`, `record_processor: 10`). ### RecordsManager Accumulates one metric record per request for the lifetime of the benchmark. The backing `ColumnStore` includes scalar metric columns, timestamp and metadata columns, categorical intern tables, and (for streaming runs) the list-valued `inter_chunk_latency` (ICL) metric. ICL is usually the dominant term when the default `ragged` backend retains every chunk gap. Source: `_estimate_records_manager`, `components.py`. For $N$ requests, $M$ standard metrics, dataset cardinality $D$, and average output length $O$: $$C = \max(1024, \text{ceil\_pow2}(N))$$ $$\text{columns} = C \times \left((M - 1 + 3 + 4) \times 8\text{B} + 6 \times 4\text{B} + 2\text{B}\right) \times 1.05$$ $$\text{intern} = (N + \min(N, D)) \times 136\text{B}$$ The column counts model the `MetricsAccumulator.process_record` layout: $M-1$ scalar metrics (the 25th standard metric is list-valued ICL), three request timestamps, four numeric metadata columns, six categorical `int32` code columns, and two boolean `uint8` columns. The categorical, boolean, and timestamp counts match `process_record` exactly. `process_record` passes five numeric metadata keys (`session_num`, `credit_issued_ns`, `request_ack_ns`, `cancellation_time_ns`, `turn_index`), but `ColumnStore.ingest_metadata` allocates a column only for non-`None` values, so the deployed column count is four for the default streaming workload — three without streaming (no `request_ack_ns`), five once any request is cancelled. `_COLUMN_STORE_METADATA_NUMERIC_COLUMNS = 4` therefore models the default, and `TestColumnStoreMetadataColumnDrift` guards it by driving records through the real `MetricsAccumulator.process_record` and counting the columns `ColumnStore` actually allocated. The intern term models a conservative request-unique `x_correlation_id` plus `conversation_id` values bounded by dataset cardinality. When streaming is enabled and `AIPERF_METRICS_LIST_BACKEND=ragged`: $$S = N \times \max(O - 1, 0)$$ $$\text{ragged} = \max(1024, \text{ceil\_pow2}(S)) \times (8 + 4)\text{B} \times 1.05 + \max(256, \text{ceil\_pow2}(N)) \times 8\text{B}$$ For $S > 0$, the first ragged term holds every ICL value as `float64` and its request index as `int32`; the second is the per-request `int64` offsets array. When $S = 0$, no ICL backend is created and the term is zero. Buffered responses also contribute no ICL storage. With `AIPERF_METRICS_LIST_BACKEND=tdigest`, the entire ICL term collapses to a bounded 4 KiB sketch (`_TDIGEST_LIST_BACKEND_BYTES`) regardless of request count, and to zero when $O \le 1$. The final variable estimate is `columns + intern + ICL + 1 MiB tracker`. Key constants: - `_FLOAT64_BYTES = 8` — numpy element width. - `_GROWABLE_ARRAY_OVERHEAD = 1.05` — wrapper-class overhead atop the numpy-backed array (also reused by GPU-telemetry and server-metrics arrays). The doubling-allocator waste is now captured separately by `ceil_pow2(N)` in the capacity term, so this multiplier only covers the ~0–2% wrapper overhead (dict of metric names, bucket tuple, sum tracker). - `_DEFAULT_NUM_STANDARD_METRICS = 25` — scalar metrics plus the one list-valued ICL metric. - `_CATEGORICAL_INTERN_BYTES_PER_REQUEST = 136` — calibrated high-cardinality string, dictionary slot, and integer-code footprint. - `ceil_pow2` rounds capacity up to the next power of two (the doubling allocator's actual footprint, not the logical request count). **Peak** applies a 10% finalization overhead on top of variable. Tracker overhead (`WorkerProcessingStats` per worker) is a flat ~1 MiB. **Warning:** if `variable_mib > 500` the estimator flags the result. For a streaming ragged run, the warning also identifies `AIPERF_METRICS_LIST_BACKEND=tdigest` as the bounded-memory alternative and notes that its ICL percentiles are approximate. **Scales with:** `total_requests` (linearly rounded up to a power of two) and `num_standard_metrics`; with ragged streaming, it also scales with `total_requests × (avg_osl_tokens - 1)`. ### DatasetManager Two memory regimes. During generation the full dataset is materialized as Pydantic `Conversation` objects; at steady state only the mmap index survives. Source: `_estimate_dataset_manager`, `components.py`. $$\text{bytes\_per\_turn} = 1500 + (\text{ISL} + \text{OSL}) \times 16$$ $$\text{peak} = \text{base} + \text{dataset\_count} \times \text{max\_turns} \times \text{bytes\_per\_turn}$$ $$\text{steady} = \text{base} + \text{dataset\_count} \times 16\text{B}$$ Key constants: - `16 bytes/token` — effective token cost after Pydantic model wrappers (~1 KiB per `Turn`), Python string headers, and the ~3x multiplier measured against ISL=100K OSL=73K with 100 entries (~297 MiB PSS). - `_MMAP_INDEX_ENTRY_BYTES = 16` — per-conversation index entry at steady state. **Scales with:** `dataset_count`, `max_turns`, `avg_isl + avg_osl`. Generation peak dominates; steady state is negligible. ### Worker One process per worker. Memory = connection pool + in-flight request records + session cache (multi-turn only). Source: `_estimate_worker`, `components.py`. Shares the `_per_request_bytes(avg_isl, avg_osl, *, streaming)` helper with RecordProcessor so the two stay consistent. Per in-flight request: $$\text{per\_request} = \underbrace{3600}_{\text{record base}} + \underbrace{(2240 + \text{ISL} \times 4)}_{\text{turn}} + \text{response}$$ Response depends on streaming mode: $$\text{response}_{\text{SSE}} = 136 + \text{OSL} \times 420 \qquad \text{response}_{\text{text}} = 152 + \text{OSL} \times 4$$ Pod-level: $$\text{variable} = \text{pool} + \text{concurrency\_per\_worker} \times \text{per\_request} + \text{session\_cache}$$ Key constants (`constants.py`): - `_REQUEST_RECORD_BASE_BYTES = 3600` — the Pydantic `RequestRecord` shell plus the `RecordContext` it carries, with empty turns/responses lists. Both are counted here because `_per_request_bytes` has no separate context term. Measured 1125 B bare record + 1896 B `RecordContext`, 3596 B for the populated pair. - `_TURN_BASE_BYTES = 2240`, `_TURN_BYTES_PER_TOKEN = 4` — `Turn` + `Text` with an empty content string measured 2235 B (`Turn` alone 1576 B, `Text` alone 592 B); the prompt text adds a measured 4.01 B/token. - `_SSE_MESSAGE_BASE_BYTES = 136`, `_SSE_BYTES_PER_CHUNK = 420` — SSE per-token cost is ~105x buffered text (420 B/chunk vs 4 B/token). The transport appends one whole `SSEMessage` per wire chunk to `RequestRecord.responses`, so the marginal cost is a message plus its packets list, its `SSEField`, and the JSON string — not a bare `SSEField`. Measured 418.7 B/chunk against the OpenAI-compatible chunk envelope this repo's mock server emits; a minimal `{"c":""}` envelope measures 286 B/chunk, so the per-chunk cost is dominated by the provider's envelope rather than by the token. - `_TEXT_RESPONSE_BASE_BYTES = 152`, `_TEXT_RESPONSE_BYTES_PER_TOKEN = 4` — the `TextResponse` `@dataclass(slots=True)` shell measured 86 B, so the base errs high by a small and deliberate margin; the body adds a measured 4.00 B/token. - `_BYTES_PER_CONNECTION = 1024` — aiohttp per-connection kernel + userspace buffers. `RequestRecord` and `Turn` are Pydantic `AIPerfBaseModel` subclasses (`extra="allow"`, so each instance carries `__dict__` + `__pydantic_extra__` + `__pydantic_fields_set__`); `TextResponse`, `SSEMessage`, and `SSEField` are `@dataclass(slots=True)` and much cheaper. The record and turn constants were originally derived assuming `msgspec.Struct` layout, which left them 2.2–5.4x low, and `_SSE_BYTES_PER_CHUNK` was fitted against a one-message-many-packets shape the transport never builds. Both were corrected on 2026-08-24; the prediction now lands at 1.01–1.07x of measured across streaming and buffered shapes at ISL=512/OSL=128 and ISL=1024/OSL=1024. Session cache activates when `max_turns > 1`: prior-turn prompts stay resident for the session duration. **Scales with:** `max_concurrency / total_workers`, `avg_osl`, streaming mode, `max_turns`, `connections_per_worker`. ### RecordProcessor One process per RP; `record_processors_per_pod` copies live in each worker pod. Memory = tokenizer cache + in-flight records + raw-batch and export-batch buffers. Source: `_estimate_record_processor`, `components.py`. $$\text{variable} = \underbrace{\text{num\_models} \times 150}_{\text{tokenizer}} + \underbrace{\text{rp\_queue\_depth} \times \text{per\_record}}_{\text{inflight}} + \text{raw\_buf} + \text{export\_buf}$$ Key constants: - `_TOKENIZER_CACHE_MIB = 150` — per distinct model (GPT-2 ~73 MiB, Llama-3 ~50–100 MiB, large SentencePiece models ~150 MiB). - `_RAW_BATCH_SIZE = 10`, `_EXPORT_BATCH_SIZE = 100`, `_EXPORT_BYTES_PER_RECORD = 1100` (module-local in `components.py`). **Queue-depth amplification at high token counts** — the `_rp_queue_depth(conc_per_rp, isl, osl)` helper in `estimator.py` models the fact that tokenization becomes the bottleneck for ISL+OSL > 10K. Records pile up in the RP's unbounded ZMQ pull queue as fully deserialized Python objects: $$\text{rp\_queue\_depth} = \begin{cases} \text{conc\_per\_rp} \times \min\!\left(\tfrac{\text{ISL}+\text{OSL}}{10000}, 10\right) & \text{if ISL}+\text{OSL} > 10{,}000 \\ \text{conc\_per\_rp} & \text{otherwise} \end{cases}$$ Calibrated against PSS: at ISL+OSL=173K the queue reaches ~150 records per RP (10x base). This is the mechanism by which large-token benchmarks OOM worker pods even at moderate concurrency. **Warnings:** triggers on `inflight_mib > 50` (high token pressure) or `tokenizer_mib > 450` (too many models loaded per RP). **Scales with:** `num_models`, `concurrency_per_rp` (amplified by token count), streaming mode, `avg_isl + avg_osl`. ### ServerMetrics Prometheus scrape history, one time series per metric per endpoint, held in growable arrays identical in shape to RecordsManager. Source: `_estimate_server_metrics`, `components.py`. $$\text{scalar\_bytes} = \text{scalar\_count} \times \text{ceil\_pow2}(\text{n\_scrapes}) \times 16\text{B}$$ $$\text{hist\_bytes} = \text{hist\_count} \times \text{ceil\_pow2}(\text{n\_scrapes}) \times (24 + \text{buckets} \times 8)\text{B}$$ $$\text{variable} = \text{num\_endpoints} \times (\text{scalar} + \text{hist} + \text{fetch}) \times 1.05$$ Key constants (`constants.py`): - `_DEFAULT_SCRAPE_INTERVAL_S = 5.0` - `_DEFAULT_UNIQUE_METRIC_SERIES = 200`, `_DEFAULT_HISTOGRAM_METRICS = 20`, `_DEFAULT_HISTOGRAM_BUCKETS = 10`. Returns zero if `num_endpoints == 0` (server metrics disabled). **Scales with:** `num_endpoints`, `duration_s / scrape_interval_s`, `unique_series`, `histogram_buckets`. ### GPUTelemetry DCGM samples held as columnar numpy arrays, one per metric per GPU. Source: `_estimate_gpu_telemetry`, `components.py`. $$\text{per\_gpu\_bytes} = \text{ceil\_pow2}(\text{n\_samples}) \times 8 + \text{num\_metrics} \times \text{ceil\_pow2}(\text{n\_samples}) \times 8$$ $$\text{variable} = \text{num\_gpus} \times \text{per\_gpu\_bytes} \times 1.05$$ Key constants: - `_DEFAULT_GPU_METRICS = 12` (DCGM default set). - `gpu_sample_interval_s = 1.0` (default in `params.py`). - `_GROWABLE_ARRAY_OVERHEAD = 1.05` — applied as a single multiplier to the total GPU telemetry footprint (same constant as RecordsManager). The `formula` string this component reports interpolates the constant, so the display and the arithmetic cannot diverge. Returns zero when `num_gpus == 0` — no DCGM URLs, or GPU telemetry disabled. The operator omits the container entirely in that case. **Scales with:** `num_gpus`, `duration_s / sample_interval_s`, `num_gpu_metrics`. ### Fixed-overhead services `SystemController`, `TimingManager`, `APIService`, `ResultsSidecar` (on the controller pod) and `WorkerGroupManager` (on each worker pod) all use `_estimate_fixed_service`: a flat `_SERVICE_BASE_MIB[name] + _PYTHON_SUBPROCESS_BASE_MIB` with `variable_mib = 0`. These do not scale with workload. Plus 3 ZMQ proxies at 5 MiB each (`_ZMQ_PROXY_MIB = 5`, `_NUM_ZMQ_PROXIES = 3`) on the controller pod. --- ## Output schema `estimate_memory(config, ...)` returns a `ClusterMemoryEstimate`: ``` ClusterMemoryEstimate ├── params: MemoryEstimationParams # echoed inputs ├── controller: PodEstimate │ ├── components: list[ComponentEstimate] # per-service breakdown │ ├── current_limit_mib: float # configured K8s limit │ ├── replicas: int │ ├── total_steady_state_mib (property) │ ├── total_peak_mib (property) │ ├── recommended_request_mib (property) # steady x 1.2 │ ├── recommended_limit_mib (property) # peak x 1.3 │ ├── headroom_pct (property) │ └── at_risk (property) # headroom < 15% ├── worker_pod: PodEstimate # single-pod, multiplied by replicas ├── operator: PodEstimate # fixed 256 MiB ├── warnings: list[str] # fixed emission order └── recommendations: list[str] # actionable tuning suggestions ``` The `replicas` field on `worker_pod` is the number of worker pods; use `worker_pod.total_steady_state_mib * worker_pod.replicas` for the cluster-wide worker footprint. Every `ComponentEstimate` also carries a `formula` string and a `dominant_factor` string that spell out how its number was derived. Neither is rendered by `format_estimate()` in `formatting.py` — the report that `aiperf kube profile` prints shows the topology header, per-pod steady/peak tables with a `[!]` marker on warned components, the cluster total, and then the `warnings` and `recommendations` lists. Read `formula` programmatically if you need the breakdown. --- ## OOM risk warnings The estimator attaches warnings at two layers. ### Per-component (attached to `ComponentEstimate.warning`) | Component | Trigger | Message pattern | |---|---|---| | RecordsManager | `variable_mib > 500` | `"variable memory is X MiB ... consider reducing request count"` | | RecordProcessor | `inflight_mib > 50` | `"in-flight records use X MiB ... driven by ISL=... OSL=... at concurrency N"` | | RecordProcessor | `tokenizer_mib > 450` | `"tokenizer cache is X MiB ... consider reducing model count"` | `ComponentEstimate.warning` holds at most one string, so the two RecordProcessor triggers are mutually exclusive: when both conditions hold, the in-flight message wins and the tokenizer message is suppressed. ### Per-cluster (appended to `ClusterMemoryEstimate.warnings`) | Warning | Trigger (constant) | Meaning | |---|---|---| | Controller pod at risk | `headroom_pct < 15%` (`_HEADROOM_WARNING_PCT`) | Peak is within 15% of limit; OOM likely. | | RecordsManager dominates controller | RM > 50% of controller limit (`_RECORDS_MANAGER_WARN_PCT`) | Request count is the single largest driver. | | Worker pod at risk | `headroom_pct < 15%` | Per-pod peak too close to limit. | | High request volume | `total_requests > 500_000` | Expect significant metric array storage. | | Many models per RP | `num_models * 150 > 450 MiB` | Tokenizer cache will dominate each RP. | | HTTP trace at scale | `export_http_trace and total_requests > 10_000` | Per-chunk trace data accumulates unboundedly. | | Multi-turn with heavy concurrency | `max_turns > 1 and sessions_per_worker > 100` | Session cache may grow significantly. | `recommendations` are built by `_build_recommendations(est)` — it prints the specific `recommended_limit_mib` values to bump to, or confirms that current limits have adequate headroom. --- ## How to run it ### Programmatic ```python import yaml from aiperf.config.config import AIPerfConfig from aiperf.kubernetes.memory_estimator import estimate_memory, format_estimate with open("bench.yaml") as fh: config = AIPerfConfig.model_validate(yaml.safe_load(fh)) est = estimate_memory( config, total_workers=10, workers_per_pod=None, # None = config.benchmark.runtime.workers_per_pod, # else AIPERF_WORKER_DEFAULT_WORKERS_PER_POD connections_per_worker=200, ) print(format_estimate(est)) if est.controller.at_risk: raise SystemExit( f"Controller would OOM: need {est.controller.recommended_limit_mib} MiB, " f"have {est.controller.current_limit_mib:.0f} MiB" ) ``` ### Via CLI `aiperf kube profile` derives `MemoryEstimationParams.from_config(...)` from the rendered config and prints the full report to **stderr** — including per-pod tables, warnings, and recommendations — before any cluster resources are created. `aiperf kube generate` runs the same estimate and prints it to **stderr** after writing the manifests to stdout. The rendered manifests take their `resources.requests` / `resources.limits` from the `AIPERF_K8S_*` resource settings, not from `recommended_request_mib` / `recommended_limit_mib` — read the report, then set the env vars yourself if it says you need to. The operator preflight step (`src/aiperf/operator/preflight/_resources.py`) runs the estimator once more as the `Memory Estimation` check. Any estimator warning downgrades that check to `WARN` and attaches `recommendations` as hints; it never blocks admission. --- ## Tuning recipes ### "My records-manager OOMs at 500k concurrency" 1. Run `aiperf kube profile` and look at the `RecordsManager` row in the Controller Pod table. 2. If `RecordsManager uses N% of controller limit` appears in warnings, read that component's `formula` string (not printed by the report — see [Output schema](#output-schema)): it separates fixed `ColumnStore` columns, categorical intern entries, and ICL storage. For streaming runs, ragged ICL scales as `requests × (OSL - 1)` and can dominate by gigabytes. 3. If exact ICL percentiles and ICL-aware sweep curves are not required, set `AIPERF_METRICS_LIST_BACKEND=tdigest`. This bounds ICL aggregation to about 4 KiB while retaining exact count/sum/min/max/average/std and approximate percentiles. Otherwise, provision for the ragged estimate. 4. Bump `AIPERF_K8S_RECORDS_MANAGER_MEMORY` (and `AIPERF_K8S_RECORDS_MANAGER_CPU`) on the operator. `current_limit_mib` for the controller pod is the **sum** of the memory limits across `CONTROLLER_RESOURCE_KEYS` (see `aiperf.kubernetes.environment`), so raise that sum until it clears the pod-level `recommended_limit_mib`. ### "Workers OOM mid-ramp on a large-token workload" Check the RecordProcessor warning first. If `in-flight records use X MiB` fires with `ISL+OSL > 10_000`, you have hit the tokenization-queue-depth amplification — at ISL+OSL=173K the queue reaches 10x `conc_per_rp`. Options: - Raise `--total-workers`. In the estimator's model `conc_per_rp` reduces to `max_concurrency / total_workers` at the default `AIPERF_K8S_RECORD_PROCESSOR_SCALE_FACTOR=1` (one RP per worker), because `workers_per_pod` scales the pod's concurrency and its RP count by the same factor — increasing `workers_per_pod` alone does not move the number. - Bump the worker pod's memory limit (`AIPERF_K8S_WORKER_POD_MEMORY`) to the estimator's `recommended_limit_mib`. - Lower `benchmark.runtime.record_processors_per_pod`. This packs fewer RPs into each pod, and the estimate follows it: at the default scale factor, dropping from one RP per worker to a single RP on a 4-worker pod removes roughly 975 MiB of peak per-pod RP memory. It raises `conc_per_rp` in exchange, so re-read the RecordProcessor warning afterwards. ### "Tokenizer cache dominates each RP" Triggered by `num_models * 150 MiB > 450 MiB`. Either split models across separate AIPerfJob CRs (one model per benchmark) or accept the higher worker-pod memory limit — there is no per-process cache deduplication. ### "The estimator disagrees with measured RSS" Constants are calibrated but static. Which class of constant is wrong decides how to fix it: - **A per-object byte constant.** Re-measure it. Extend `TestPerRequestBytesAgainstMeasuredHeap` in `tests/unit/kubernetes/test_memory_estimator.py` with your shape, read the measured value out of the assertion message, and update the constant. The test enforces `1.0 <= predicted / measured <= 1.6`, so a prediction that lands *below* measured is a bug and a prediction slightly above it is correct — the output is a limit recommendation. The per-chunk SSE cost is dominated by the provider's chunk envelope, so a server with unusually large chunks (per-chunk `usage`, `logprobs`) will legitimately exceed the model. - **An aggregate MiB baseline.** These come from a real-cluster `container_memory_working_set_bytes` sweep and cannot be re-derived in-process. Update them only from a new cluster measurement. Record the provenance in the constant's comment the way the existing ones do — state the method, the date, and the shape measured. Do not edit the formulas ad-hoc. --- ## References - Public API: `src/aiperf/kubernetes/memory_estimator.py` - Orchestrator: `src/aiperf/kubernetes/_memory_estimator/estimator.py` - Per-component formulas: `src/aiperf/kubernetes/_memory_estimator/components.py` - Calibration constants: `src/aiperf/kubernetes/_memory_estimator/constants.py` - Result dataclasses: `src/aiperf/kubernetes/_memory_estimator/estimates.py` - Param extraction: `src/aiperf/kubernetes/_memory_estimator/params.py` - Formatter: `src/aiperf/kubernetes/_memory_estimator/formatting.py`