Skip to navigation

Model Pricing

View as MarkdownOpen in Claude

Use this reference for catalog and resolver contracts and normalized token and cost fields. For configuration and verification, refer to Model Pricing. For response-codec integration, refer to Provider Response Codecs and Model Pricing.

Cost Estimation

Model pricing estimates carry pricing_provider, pricing_model, pricing_as_of, pricing_source, and currency metadata so stale model pricing can be audited without failing response decoding. Normalized cost uses currency-neutral amount fields such as total, input, output, cache_read, and cache_write. Unknown model pricing and missing token data are non-fatal: Relay omits the cost field and still exports token metrics and response annotations.

Relay resolves model pricing through an active PricingResolver source chain. Provider or framework-reported cost remains authoritative. The resolver runs only when Usage.cost is missing. Relay does not ship provider price data by default: estimates require a configured inline, file, or embedding-provided model pricing source. With no configured source, every model is treated as unknown for model pricing.

Each catalog entry declares:

  • provider and canonical model_id.
  • aliases for dated or provider-specific model IDs.
  • currency, defaulting to USD.
  • unit, defaulting to per_token. Relay estimates only per_token entries in this version. per_request, per_second, and gpu_hour are representable for future source integrations but are not estimated.
  • rates per one million input, output, cache-read, and cache-write tokens, with cache-write rates per cache duration, for flat per_token entries.
  • rate_schedule for data-driven threshold-based model pricing, such as models whose full-request input/output rates change after a prompt-token threshold.
  • prompt_cache.read_accounting to tell Relay whether prompt tokens include cache-read tokens. Relay ignores it for usage with uncached_input_tokens.
  • pricing_as_of and pricing_source for auditability.

Relay validates catalogs at startup and rejects duplicate canonical IDs or aliases within the same normalized provider/model key. The same model ID can appear under distinct providers, such as openai/gpt-4o-mini and azure/openai/gpt-4o-mini. Adding a model should be a catalog/source update plus tests. It should not require adding another Rust match arm.

Model lookup is provider-aware and route-aware. Relay uses the managed LLM call name as the provider/route and first tries provider-scoped keys for the full model and terminal model name, then falls back to model-only suffixes. For example, a call named azure/openai with response model = "gpt-4o-mini" tries azure/openai/gpt-4o-mini before generic gpt-4o-mini. If the model string is itself routed, such as azure/openai/gpt-4o-mini, Relay can infer azure/openai for the terminal model before trying slash-delimited model-only suffixes. This keeps route-specific enterprise model pricing authoritative when configured while still allowing generic model pricing to apply to routed names.

For threshold-based model pricing, use rate_schedule.type = "prompt_token_threshold". Relay picks exactly one tier from the request’s input tokens, including cache reads and writes. It applies that tier to the full request. It does not price only the overflow tokens at the higher rate. This matches providers that publish “short context” and “long context” prices for the entire request/session. If usage has neither prompt_tokens nor uncached_input_tokens for a thresholded entry, Relay omits the estimate instead of guessing. Counts that the estimate folds in from api_specific, such as Gemini tool-use prompt tokens, are priced at the selected tier’s rates.

The following catalog entry applies one pricing tier to the full request.

Example catalog entry
{
"provider": "google",
"model_id": "gemini-3.1-pro-preview",
"aliases": ["gemini-3.1-pro-preview-customtools"],
"pricing_as_of": "2026-06-05",
"pricing_source": "https://ai.google.dev/gemini-api/docs/pricing",
"rate_schedule": {
"type": "prompt_token_threshold",
"applies_to": "full_request",
"tiers": [
{
"max_prompt_tokens": 200000,
"rates": {
"input_per_million": 2.0,
"output_per_million": 12.0,
"cache_read_per_million": 0.2
}
},
{
"min_prompt_tokens": 200001,
"rates": {
"input_per_million": 4.0,
"output_per_million": 18.0,
"cache_read_per_million": 0.4
}
}
]
},
"prompt_cache": {
"read_accounting": "included_in_prompt_tokens"
}
}

Expected result: With no cache usage and 1,000 output tokens, the rates in this example produce the following totals. These results use the example rates, not a live provider price lookup.

Input TokensExpected Total (USD)
200,0000.412
200,0010.818004

Database-backed or remote model pricing should be implemented as a source that returns a validated PricingCatalog snapshot to Relay. Keep database queries, service auth, refresh cadence, and caching outside the LLM response hot path. A fleet deployment can refresh /etc/nemo-relay/pricing.json on Unix or the corresponding %ProgramData%\nemo-relay\pricing.json file on Windows from an IT-managed service, or embed a custom Rust PricingSource that reads from a database and installs a PricingResolver snapshot during process startup.

External model pricing catalogs should be converted into Relay catalog JSON out-of-band and then loaded through a file source, unless the embedding application installs a custom Rust PricingSource directly.

Token and Cost Field Semantics

This section is the stable reference for the token and cost fields on LLM end events, carried on AnnotatedLlmResponse.usage.

Granularity

Every token and cost value is per LLM call (one provider completion) unless it is an explicit aggregate:

  • AnnotatedLlmResponse.usage — The single LLM end event it is attached to.
  • OpenTelemetry projection attributes — The single LLM span they appear on; spans are never summed across calls.
  • ATIF steps — An exported LLM call typically yields a user start step (no metrics) and an agent end step that carries the call’s metrics.
  • ATIF final_metrics.total_* — The only aggregate: a per-trajectory sum of the metric fields present on its steps. It excludes embedded subagent trajectories and can be partial.

No exporter emits a running cross-call total other than ATIF final_metrics.

Usage Fields

All normalized fields are optional. A provider can omit a field, while a codec can compute one (such as Anthropic’s total_tokens) and configured pricing can synthesize cost. Usage has no catch-all field, so provider usage fields that Relay does not model are dropped.

FieldMeaning
prompt_tokensInput/prompt tokens.
completion_tokensOutput/completion tokens, passed through unmodified. For OpenAI Responses this is the provider’s output_tokens (which per OpenAI already includes any reasoning tokens); the reasoning count is reported separately under api_specific and is not added on top.
total_tokensProvider-reported, or computed as prompt + completion by some codecs (such as Anthropic) when the provider omits it.
cache_read_tokensPrompt-cache read tokens, when the provider reports prompt caching.
cache_write_tokensPrompt-cache write tokens as a list of {count, duration}, one entry per cache duration in seconds. duration is omitted when the provider does not say.
uncached_input_tokensInput tokens without cache reads or cache writes. When present, pricing bills this count at the input rate.
costNormalized CostEstimate, when reported by the provider or estimable from configured pricing.

Built-in codecs normalize provider field names as follows:

Normalized FieldOpenAI ChatOpenAI ResponsesAnthropic MessagesOCI Generative AIGemini generateContent
prompt_tokensprompt_tokensinput_tokensinput_tokenspromptTokenspromptTokenCount
completion_tokenscompletion_tokensoutput_tokensoutput_tokenscompletionTokenscandidatesTokenCount
total_tokenstotal_tokenstotal_tokenscomputedtotalTokenstotalTokenCount, or computed as prompt + candidates + tool-use prompt + thinking
cache_read_tokensprompt_tokens_details.cached_tokensinput_tokens_details.cached_tokenscache_read_input_tokenspromptTokensDetails.cachedTokenscachedContentTokenCount
cache_write_tokens——cache_creation_input_tokens, split into 300 and 3600 second entries by cache_creation.ephemeral_1h_input_tokens when present——
uncached_input_tokensprompt_tokens - prompt_tokens_details.cached_tokens, when both are reportedinput_tokens - input_tokens_details.cached_tokens, when both are reportedinput_tokenspromptTokens - promptTokensDetails.cachedTokens, when both are reportedpromptTokenCount - cachedContentTokenCount, when both are reported
costcost_usd, structured cost, or scalar cost (OpenRouter; USD provider-reported total)cost_usd, structured cost, or scalar cost (USD provider-reported total)cost_usd, structured cost, or scalar cost (USD provider-reported total)——

Gemini generateContent thinking tokens (thoughtsTokenCount) are stored in api_specific.thoughts_tokens rather than completion_tokens, because Google bills them as output tokens but reports them separately. The cost estimate folds them into the effective output-token count so that the pricing table reflects the real billing cost.

Gemini tool-use prompt tokens, reported as toolUsePromptTokenCount, are stored in api_specific.tool_use_prompt_tokens and counted in a computed total_tokens. Relay bills those tokens at the input rate for URL context, code execution, and File Search. Relay excludes them for Google Search, image search, and Google Maps grounding, which are billed per request. When the count is nonzero and the tool evidence is mixed, unrecognized, or missing, Relay leaves the cost estimate unset. Mixed evidence includes Search together with URL context or code execution. Unrecognized evidence includes retrievedContext without a non-empty fileSearchStore, such as Vertex AI Search. When the count is absent or 0, the estimate is unaffected.

Relay applies both rules to any response annotation that reports thoughts_tokens or tool_use_prompt_tokens through the gemini_generate_content api_specific variant, using grounding_metadata, urlContextMetadata, and Gemini code execution parts to identify tool usage.

Built-in codecs preserve only modeled provider-specific usage details under api_specific; other usage fields are dropped. For example, OpenAI Responses reasoning token counts are kept under api_specific (output_tokens_details), but OpenAI Chat completion_tokens_details is not.

Cost Fields

CostEstimate carries cost amounts (in currency) plus pricing provenance. Refer to Cost Estimation above for resolution order and pricing setup.

FieldMeaning
totalOptional total cost in currency. When absent, some exporters derive a total from the component amounts.
currencyISO 4217 code; defaults to USD.
input / output / cache_read / cache_writePer-category amounts in currency.
sourceprovider_reported (authoritative) or model_pricing (estimated).
pricing_provider / pricing_model / pricing_as_of / pricing_sourceEstimate provenance, for auditing stale pricing.

For Chat Completions, Responses, and Messages, cost_usd takes precedence over usage.cost. Unrecognized usage.cost values are ignored without discarding token usage or other response annotations; configured pricing can still supply an estimate.

Missing is not zero: an absent cost or token field means unknown, while an explicit 0 is a reported value and is preserved. Relay does not convert currencies.

Exporter Field Mapping

Each exporter projects usage/cost differently. Projections do not change the canonical fields above.

MeasurementATOFATIF step / final_metricsopeninferencefullgen_ai
Prompt tokensfull usage preservedprompt_tokens / total_prompt_tokensllm.token_count.promptnot emittedgen_ai.usage.input_tokens
Completion tokenspreservedcompletion_tokens / total_completion_tokensllm.token_count.completionnot emittedgen_ai.usage.output_tokens
Total tokenspreservedno first-class fieldllm.token_count.totalnot emittednot emitted
Cache read / writepreservedsummed into cached_tokens / total_cached_tokensllm.token_count.prompt_details.cache_read / …cache_writenot emittedgen_ai.usage.cache_read.input_tokens / gen_ai.usage.cache_write.input_tokens
Costfull cost preservedcost_usd / total_cost_usd (USD only)llm.cost.total (USD only)nemo_relay.llm.cost.total + nemo_relay.llm.cost.currency (any currency)not emitted

The full OpenTelemetry projection carries cost in any currency, while ATIF and the openinference projection report cost only when it is USD-denominated and otherwise omit it. The gen_ai projection emits standardized token attributes but no cost attributes. ATIF derives metrics from codec-normalized usage where available and fills missing supported fields from the raw payload. metrics.extra holds only unmapped keys from the raw usage/token_usage object (for example reasoning token counts, or a raw total_tokens), and only when the step already has a recognized metric. Normalized-only or total-only values are not projected.

Stability

The Usage and CostEstimate field names and meanings, and the exporter mappings above, are stable as of ATOF 0.1 (ATIF schema ATIF-v1.7, pricing catalog version: 1). Future NeMo Relay releases can add new optional fields to the serialized JSON/ATOF shapes. Renames or removals are breaking changes and are called out in release notes.

The Rust Usage and CostEstimate structs and the CostSource enum are exhaustive, so adding a field or variant is a source-breaking change for Rust consumers.

The following behaviors are intentional in this release but can change later:

  • The full OpenTelemetry projection emits cost only, not token counts.
  • The gen_ai OpenTelemetry projection emits token counts but not cost.
  • ATIF and the openinference projection report cost only in USD.
  • Reasoning tokens are not a first-class Usage field.
  • Bindings expose usage/cost as snake_case JSON rather than typed objects.