Skip to navigation
Use PluginsBuilt-in Plugins

Model Pricing

View as Markdown

Use the built-in pricing component to configure model pricing catalogs for cost estimates on managed or manually observed LLM responses. Manual lifecycle end events must supply a response codec or normalized response annotation with model and token usage. Relay leaves cost absent when no configured source matches the provider, model, and token usage.

Configuration

Add the pricing component to plugins.toml. The following configuration loads a JSON catalog and defines an inline catalog. Relay searches sources in the listed order, so place overrides before their fallbacks:

version = 1
[[components]]
kind = "pricing"
enabled = true
[[components.config.sources]]
type = "file"
path = "./pricing.json"
[[components.config.sources]]
type = "inline"
[components.config.sources.catalog]
version = 1
[[components.config.sources.catalog.entries]]
provider = "openai"
model_id = "gpt-4o-mini"
aliases = ["openai/openai/gpt-4o-mini"]
currency = "USD"
unit = "per_token"
pricing_as_of = "2026-06-06"
pricing_source = "internal-pricing-snapshot"
[components.config.sources.catalog.entries.rates]
input_per_million = 0.15
output_per_million = 0.60
cache_read_per_million = 0.075
[components.config.sources.catalog.entries.prompt_cache]
read_accounting = "included_in_prompt_tokens"

Use type = "file" with a JSON catalog path or type = "inline" with the catalog in plugins.toml. Relay checks sources and catalog entries in listed order, and it uses the first entry that matches the provider and model.

Relay publishes a numeric total only when every nonzero usage category has a configured rate. If a response reports nonzero cache-read or cache-write usage and the matching catalog entry omits cache_read_per_million or cache_write_per_million, Relay keeps the priced components but omits the aggregate total. Set a rate to 0.0 when a category is intentionally free.

Some usage reports uncached_input_tokens, a count of input tokens without cache reads or cache writes. Relay picks the input count to bill as follows:

  • When usage has uncached_input_tokens, Relay bills that count at the input rate and ignores read_accounting. If usage also has prompt_tokens, that count must equal uncached_input_tokens or uncached_input_tokens plus the cache reads and writes. Otherwise, Relay leaves the usage unpriced.
  • For other usage, read_accounting tells Relay whether prompt_tokens includes cache reads. When prompt_tokens includes them, Relay subtracts them before billing. If cache reads are larger than prompt_tokens, Relay leaves the usage unpriced.

Relay bills cache reads and writes at their own rates. To pick a threshold tier, Relay counts all input tokens, including cache reads and writes.

When Relay merges explicit-or-user and system configuration files, it prepends higher-priority sources instead of replacing lower-priority sources. The effective order is system, then explicit-or-user. This lets an enterprise catalog override narrower catalogs while preserving those catalogs as fallbacks.

Manage Catalog Sources with the CLI

Run the following commands to validate a catalog and add it to user configuration:

nemo-relay model-pricing validate <path/to/pricing.json>
nemo-relay model-pricing init --user
nemo-relay model-pricing add-source <path/to/pricing.json> --user

Replace <path/to/pricing.json> with the path to your pricing catalog before running either command. Use --user for the user configuration file or --global for system configuration at /etc/nemo-relay/plugins.toml on Unix or %ProgramData%\nemo-relay\plugins.toml on Windows. add-source places a new source ahead of existing sources by default. Use --append to keep it as a fallback.

Verify the Active Configuration

Run the following command to resolve a model against the configured catalogs:

nemo-relay model-pricing resolve gpt-4o-mini \
  --provider openai \
  --prompt-tokens 1000 \
  --completion-tokens 500

Use nemo-relay doctor to validate configured model pricing sources with the rest of the gateway configuration. Refer to Add Model Pricing for Cost Estimates for the catalog schema, source precedence, and complete CLI workflow.