Model Pricing
Use this reference for catalog and resolver contracts and normalized token and cost fields. For configuration and verification, refer to Model Pricing. For response-codec integration, refer to Provider Response Codecs and Model Pricing.
Cost Estimation
Model pricing estimates carry pricing_provider, pricing_model,
pricing_as_of, pricing_source, and currency metadata so stale model
pricing can be audited without failing response decoding. Normalized cost uses
currency-neutral amount fields such as total, input, output,
cache_read, and cache_write.
Unknown model pricing and missing token data are non-fatal: Relay omits the cost
field and still exports token metrics and response annotations.
Relay resolves model pricing through an active PricingResolver source chain.
Provider or framework-reported cost remains authoritative. The resolver runs
only when Usage.cost is missing. Relay does not ship provider price data
by default: estimates require a configured inline, file, or embedding-provided
model pricing source. With no configured source, every model is treated as
unknown for model pricing.
Each catalog entry declares:
providerand canonicalmodel_id.aliasesfor dated or provider-specific model IDs.currency, defaulting toUSD.unit, defaulting toper_token. Relay estimates onlyper_tokenentries in this version.per_request,per_second, andgpu_hourare representable for future source integrations but are not estimated.ratesper one million input, output, cache-read, and cache-write tokens, with cache-write rates per cache duration, for flatper_tokenentries.rate_schedulefor data-driven threshold-based model pricing, such as models whose full-request input/output rates change after a prompt-token threshold.prompt_cache.read_accountingto tell Relay whether prompt tokens include cache-read tokens. Relay ignores it for usage withuncached_input_tokens.pricing_as_ofandpricing_sourcefor auditability.
Relay validates catalogs at startup and rejects duplicate canonical IDs or
aliases within the same normalized provider/model key. The same model ID can
appear under distinct providers, such as openai/gpt-4o-mini and
azure/openai/gpt-4o-mini. Adding a model should be a catalog/source update
plus tests. It should not require adding another Rust match arm.
Model lookup is provider-aware and route-aware. Relay uses the managed LLM call
name as the provider/route and first tries provider-scoped keys for the full
model and terminal model name, then falls back to model-only suffixes. For
example, a call named azure/openai with response model = "gpt-4o-mini" tries
azure/openai/gpt-4o-mini before generic gpt-4o-mini. If the model string is
itself routed, such as azure/openai/gpt-4o-mini, Relay can infer
azure/openai for the terminal model before trying slash-delimited model-only
suffixes. This keeps route-specific enterprise model pricing authoritative when
configured while still allowing generic model pricing to apply to routed names.
For threshold-based model pricing, use rate_schedule.type = "prompt_token_threshold".
Relay picks exactly one tier from the request’s input tokens, including cache
reads and writes. It applies that tier to the full request. It does not price
only the overflow tokens at the higher rate. This matches providers that publish
“short context” and “long context” prices for the entire request/session. If
usage has neither prompt_tokens nor uncached_input_tokens for a thresholded
entry, Relay omits the estimate instead of guessing. Counts that the estimate
folds in from api_specific, such as Gemini tool-use prompt tokens, are priced
at the selected tier’s rates.
The following catalog entry applies one pricing tier to the full request.
Expected result: With no cache usage and 1,000 output tokens, the rates in this example produce the following totals. These results use the example rates, not a live provider price lookup.
Database-backed or remote model pricing should be implemented as a source that
returns a validated PricingCatalog snapshot to Relay. Keep database queries,
service auth, refresh cadence, and caching outside the LLM response hot path. A
fleet deployment can refresh /etc/nemo-relay/pricing.json on Unix or the
corresponding %ProgramData%\nemo-relay\pricing.json file on Windows from an
IT-managed service, or embed a custom Rust PricingSource that reads from a
database and installs a PricingResolver snapshot during process startup.
External model pricing catalogs should be converted into Relay catalog JSON
out-of-band and then loaded through a file source, unless the embedding
application installs a custom Rust PricingSource directly.
Token and Cost Field Semantics
This section is the stable reference for the token and cost fields on LLM end
events, carried on AnnotatedLlmResponse.usage.
Granularity
Every token and cost value is per LLM call (one provider completion) unless it is an explicit aggregate:
AnnotatedLlmResponse.usage— The single LLM end event it is attached to.- OpenTelemetry projection attributes — The single LLM span they appear on; spans are never summed across calls.
- ATIF steps — An exported LLM call typically yields a
userstart step (no metrics) and anagentend step that carries the call’smetrics. - ATIF
final_metrics.total_*— The only aggregate: a per-trajectory sum of the metric fields present on its steps. It excludes embedded subagent trajectories and can be partial.
No exporter emits a running cross-call total other than ATIF final_metrics.
Usage Fields
All normalized fields are optional. A provider can omit a field, while a codec can
compute one (such as Anthropic’s total_tokens) and configured pricing can
synthesize cost. Usage has no catch-all field, so provider usage fields that
Relay does not model are dropped.
Built-in codecs normalize provider field names as follows:
Gemini generateContent thinking tokens (thoughtsTokenCount) are stored in api_specific.thoughts_tokens rather than completion_tokens, because Google bills them as output tokens but reports them separately. The cost estimate folds them into the effective output-token count so that the pricing table reflects the real billing cost.
Gemini tool-use prompt tokens, reported as toolUsePromptTokenCount, are stored in api_specific.tool_use_prompt_tokens and counted in a computed total_tokens. Relay bills those tokens at the input rate for URL context, code execution, and File Search. Relay excludes them for Google Search, image search, and Google Maps grounding, which are billed per request. When the count is nonzero and the tool evidence is mixed, unrecognized, or missing, Relay leaves the cost estimate unset. Mixed evidence includes Search together with URL context or code execution. Unrecognized evidence includes retrievedContext without a non-empty fileSearchStore, such as Vertex AI Search. When the count is absent or 0, the estimate is unaffected.
Relay applies both rules to any response annotation that reports thoughts_tokens or tool_use_prompt_tokens through the gemini_generate_content api_specific variant, using grounding_metadata, urlContextMetadata, and Gemini code execution parts to identify tool usage.
Built-in codecs preserve only modeled provider-specific usage details under
api_specific; other usage fields are dropped. For example, OpenAI Responses
reasoning token counts are kept under api_specific (output_tokens_details),
but OpenAI Chat completion_tokens_details is not.
Cost Fields
CostEstimate carries cost amounts (in currency) plus pricing provenance. Refer to
Cost Estimation above for resolution order and pricing setup.
For Chat Completions, Responses, and Messages, cost_usd takes precedence over
usage.cost. Unrecognized usage.cost values are ignored without discarding
token usage or other response annotations; configured pricing can still supply
an estimate.
Missing is not zero: an absent cost or token field means unknown, while an
explicit 0 is a reported value and is preserved. Relay does not convert
currencies.
Exporter Field Mapping
Each exporter projects usage/cost differently. Projections do not change the
canonical fields above.
The full OpenTelemetry projection carries cost in any currency, while ATIF
and the openinference projection report cost only when it is USD-denominated
and otherwise omit it. The gen_ai projection emits standardized token
attributes but no cost attributes. ATIF derives metrics from codec-normalized
usage where available and fills missing supported fields from the raw payload.
metrics.extra holds only unmapped keys from the raw
usage/token_usage object (for example reasoning token counts, or a raw
total_tokens), and only when the step already has a recognized metric.
Normalized-only or total-only values are not projected.
Stability
The Usage and CostEstimate field names and meanings, and the exporter
mappings above, are stable as of ATOF 0.1 (ATIF schema ATIF-v1.7, pricing
catalog version: 1). Future NeMo Relay releases can add new optional fields to
the serialized JSON/ATOF shapes. Renames or removals are breaking changes and are
called out in release notes.
The Rust Usage and CostEstimate structs and the CostSource enum are
exhaustive, so adding a field or variant is a source-breaking change for Rust
consumers.
The following behaviors are intentional in this release but can change later:
- The
fullOpenTelemetry projection emits cost only, not token counts. - The
gen_aiOpenTelemetry projection emits token counts but not cost. - ATIF and the
openinferenceprojection report cost only in USD. - Reasoning tokens are not a first-class
Usagefield. - Bindings expose
usage/costas snake_case JSON rather than typed objects.