# Normalizing token usage across providers - api.llmnet.nl

[Skip to content](#lm-inhoud)Network/[NL](/en/token-usage-normalisatie-providers)EN[Hubhub.llmnet.nlCompare models on task, language, cost and license.](https://hub.llmnet.nl/en/)[Communitycommunity.llmnet.nlPrompt techniques, patterns and system prompts.](https://community.llmnet.nl/en/)[APIapi.llmnet.nlLLMs in production: rate limits, routing, structured output.](https://api.llmnet.nl/en/)[Consultancyconsultancy.llmnet.nlRolling out AI in an organization, pilot to production.](https://consultancy.llmnet.nl/en/)[Newsnieuws.llmnet.nlAI developments, explained for the Netherlands.](https://nieuws.llmnet.nl/en/)[Benchmarkbenchmark.llmnet.nlMeasure AI quality yourself, on your own tasks.](https://benchmark.llmnet.nl/en/)[Careersvacatures.llmnet.nlAI roles, salaries and career paths in the Netherlands.](https://vacatures.llmnet.nl/en/)[Learnleren.llmnet.nlAI concepts in plain language, beginner to builder.](https://leren.llmnet.nl/en/)[Guidegids.llmnet.nlRun AI privately on your own Mac, PC, NAS or home server.](https://gids.llmnet.nl/en/)[Directorydirectory.llmnet.nlMapping the AI ecosystem: tools, models, companies.](https://directory.llmnet.nl/en/)[Radarradar.llmnet.nlSignals from X, research and communities for indie developers.](https://radar.llmnet.nl/en/)[Appsapps.llmnet.nlReviews of AI apps and open-source repos, with tips for builders.](https://apps.llmnet.nl/en/)[llmnet.nl — main site](https://llmnet.nl/en/)[](https://x.com/intent/post?url=https%3A%2F%2Fapi.llmnet.nl%2Fen%2Ftoken-usage-normalisatie-providers&text=Normalizing%20token%20usage%20across%20providers%20-%20api.llmnet.nl)[](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fapi.llmnet.nl%2Fen%2Ftoken-usage-normalisatie-providers)[](https://www.reddit.com/submit?url=https%3A%2F%2Fapi.llmnet.nl%2Fen%2Ftoken-usage-normalisatie-providers&title=Normalizing%20token%20usage%20across%20providers%20-%20api.llmnet.nl)[](#)[](https://x.com/intent/post?url=https%3A%2F%2Fapi.llmnet.nl%2Fen%2Ftoken-usage-normalisatie-providers&text=Normalizing%20token%20usage%20across%20providers%20-%20api.llmnet.nl)[](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fapi.llmnet.nl%2Fen%2Ftoken-usage-normalisatie-providers)[](https://www.reddit.com/submit?url=https%3A%2F%2Fapi.llmnet.nl%2Fen%2Ftoken-usage-normalisatie-providers&title=Normalizing%20token%20usage%20across%20providers%20-%20api.llmnet.nl)[](#)

# Normalizing token usage across providers

By Ivo Donker — compiled with AI assistance (Claude & Gemini) · Last updated: August 6, 2026

Organizations deploying multiple Large Language Models (LLMs) across different API providers quickly run into a hard reality: the number of reported tokens is not a universal unit of measurement. A token with provider A does not equal a token with provider B, neither in information content, technical handling, nor cost structure. Building a reliable metrics layer that provides insight into efficiency, budgets, and usage patterns requires a well-thought-out normalization architecture.

In this article, we cover the technical and conceptual challenges of normalizing token usage across various AI providers. We discuss how tokenizers work, isolating diverging API response fields, the specific handling of reasoning and cache tokens, and designing a future-proof data model.

## Why a token is not a universal unit

The idea that a token represents a fixed amount of text or information is a misconception. A token is simply the output of a specific algorithm (the tokenizer) that splits text into numeric building blocks for a specific neural network. Because each provider maintains its own vocabulary and splitting rules, the information density per token varies considerably.

How these algorithms work is discussed in detail in the overview of [tokenization explained](https://leren.llmnet.nl/en/tokenisatie-uitgelegd). For application architects, this means that two different models can register a completely different number of input tokens for the exact same input text.

### The systematic disadvantage for the Dutch language

This phenomenon has a direct impact on multilingual applications, and particularly on the Dutch language. Most commercial tokenizers are primarily trained on English datasets and source code. As a result, the tokenizer's vocabulary table predominantly contains complete English words or common English word fragments.

When Dutch text is processed by such a tokenizer, fragmentation occurs. Common Dutch words, compound nouns, and specific conjugations are split into numerous separate syllables or even individual characters. A Dutch paragraph can therefore require up to 40% to 70% more tokens than the exact same content in English. When building applications that switch between models from different vendors, changing a model version immediately shifts the measured token volume, even if the underlying text flow remains identical.

## Fragmentation across API response fields

Beyond the intrinsic differences in tokenization, API providers utilize divergent data structures in their JSON responses. Where the basic distinction once merely consisted of input tokens (prompt) and output tokens (completion), the spectrum of fields has since grown significantly more complex.

In practice, you will encounter the following field categories in API responses, among others:

- Standard input (input / prompt tokens): The tokens processed to interpret the prompt.

- Standard output (output / completion tokens): The generated text tokens that are visible in the response.

- Cache-read input: Input tokens retrieved from an existing context cache, which are billed at a discounted rate.

- Cache-write input: Input tokens used to build a new context cache.

- Hidden reasoning tokens: Internal thought steps in reasoning models (reasoning or thought tokens) that are billed, but are not returned to the client as visible text.

Simply combining these numbers into a single 'total token count' makes any form of cost and performance analysis impossible.

## Categorizing reasoning tokens and cache tokens

Correctly handling specialized token types requires clear logic during the collection and analysis phase. Two categories require special attention: reasoning tokens and cache tokens.

### Separating reasoning tokens from visible output

Models with advanced reasoning capabilities generate internal thought steps before formulating their final response. These 'reasoning tokens' are billed by the API at the output token rate. However, they do not appear in the final text field of the API response.

It is crucial to strictly separate these thought steps from regular output tokens in your data model. Lumping reasoning tokens together with generated text pollutes your performance metrics. For instance, you will no longer be able to accurately calculate how many output tokens per second your application produces for the user, or what the average length of the actually displayed answers is. Therefore, always track reasoning tokens as a separate metric field.

### Cache tokens: writing versus reading

Context caching allows developers to store large volumes of static information (such as documentation or system instructions) in the provider's memory. However, the financial and technical handling of cache tokens has two sides:

- Cache creation (write): The one-time processing and writing of the context to the cache. This often requires more processing power and can sometimes be billed at a separate rate.

- Cache usage (read): Reusing the cached context in follow-up requests. This reduces processing time and is billed at a heavily reduced rate.

Some providers deduct cached tokens from the total number of input tokens, while others provide a total number that includes cached tokens as a subfield. In your normalization layer, you must parse the API payloads in such a way that both 'gross input' and 'cache hits' are stored unambiguously.

## Separating internal calculation units from raw data

A fundamental design principle when building an LLM measurement layer is separating measured facts from calculated assumptions. You should never confuse an estimated or converted number with a hard, measured value.

Design rule: Always preserve the provider's raw API response information in its original form. In addition, store normalized or derived values in separate column structures.

If a provider reports that a request consumed 1,200 `prompt_tokens`, store that exact number in the raw input field. To enable comparisons across different models, you can also calculate an internal normalized unit, for example based on a standard character length or a reference tokenizer. Should the calculation method of your internal unit change in the future, the unaltered raw data will always allow you to perform historical recalculations.

## Decoupling costs from token counts

Directly storing a calculated monetary amount based on hardcoded variables in the application code is a common pitfall. Prices per million tokens change regularly, and providers continuously introduce new discount structures or peak and off-peak rates. Anyone looking to read more about the diverse structure of rates can consult the article on [per-token pricing models](https://hub.llmnet.nl/en/prijsmodellen-per-token-uitgelegd) .

To prevent price changes from rewriting your historical financial reporting, you should decouple cost calculations from token volume. You achieve this by maintaining a decoupled pricing table (pricing matrix) with validity periods.

Model ID | 
Token Category | 
Price per 1M (EUR) | 
Valid From | 
Valid To | 

model-alpha-v1 | 
input_standard | 
No fixed price | 
2026-01-01 | 
2026-06-30 | 

model-alpha-v1 | 
input_standard | 
New rate | 
2026-07-01 | 
NULL | 

model-alpha-v1 | 
input_cached | 
Reduced rate | 
2026-01-01 | 
NULL | 

model-alpha-v1 | 
output_reasoning | 
Output rate | 
2026-01-01 | 
NULL | 

When processing a request, link it to the price version that is valid at that moment (`price_version_id`). This ensures that the historical cost calculation of a request from May 2026 remains strictly identical, even if the rate for that specific model is adjusted in July 2026.

## Handling missing or incomplete usage data

In an ideal world, every API call returns a neatly formatted JSON object with an exact summary of the usage. In practice, data loss occurs regularly, particularly when using streaming responses (Server-Sent Events).

When a user closes the browser while streaming, or when a network disruption drops the connection before the concluding usage chunk is received, the provider's official metrics will be missing. However, the provider has incurred costs for the processed input and the partially generated output at that point.

### Estimating with a custom tokenizer

To prevent gaps in your data model, your application must fall back on a local estimation module in such cases. For this, use a local tokenizer library to count the submitted prompt and the text elements received up to that point.

It is crucial that the outcome of such a calculation is explicitly flagged. Use a boolean flag such as `is_estimated = true` or a status field like `usage_source = 'local_fallback'`. This prevents estimated data from accidentally being mistaken for actual billing data during later audits.

## The data model for a practical telemetry layer

To store all raw data, calculated costs, and functional context in a structured manner, a robust database schema is required. The SQL table structure below illustrates how a normalized metrics table can be set up.

CREATE TABLE llm_usage_logs (
 request_id UUID PRIMARY KEY,
 created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
 provider VARCHAR(50) NOT NULL,
 model_version VARCHAR(100) NOT NULL,
 
 -- Ruwe metingen van provider
 raw_input_tokens INT DEFAULT 0,
 raw_output_tokens INT DEFAULT 0,
 raw_cached_input_tokens INT DEFAULT 0,
 raw_reasoning_tokens INT DEFAULT 0,
 
 -- Status van de meting
 is_estimated BOOLEAN DEFAULT FALSE,
 usage_source VARCHAR(20) DEFAULT 'provider_api',
 
 -- Financiële koppeling
 price_version_id INT REFERENCES price_matrix(id),
 calculated_cost_eur NUMERIC(10, 6),
 
 -- Functionele attributie
 calling_function VARCHAR(100) NOT NULL,
 team_id VARCHAR(50),
 user_hash VARCHAR(64)
);

This table serves as the fundamental measurement layer beneath your financial administration. For allocating these figures to organizational units, see the article on [allocating costs per user](https://api.llmnet.nl/en/kosten-per-gebruiker-toerekenen), which builds upon the logic described here.

## Attribution and privacy: allocation without content

For auditing and internal chargebacks, it is essential to know which user, team, or specific software feature is responsible for which portion of token consumption. At the same time, privacy regulations (such as the GDPR) and internal security guidelines impose strict requirements on storing personal data and prompt contents.

The metrics layer must therefore remain strictly separated from the substantive data stream:

- Never store prompt content in the metrics table: Do not store user inputs, generated outputs, or embedded documents in your token consumption logging.

- Pseudonymize user IDs: Do not store direct email addresses or names. Use cryptographic hashes (such as SHA-256 with an internal salt) to link requests to an anonymous identity.

- Store functional metadata: Record which microservice, API endpoint, or specific background job initiated the call (`calling_function`).

By adopting this approach, you can generate detailed reports on consumption patterns and costs per department without turning the logging system into a privacy-sensitive data source. This method aligns seamlessly with the guidelines for [observability and logging](https://api.llmnet.nl/en/observability-en-logging) within AI architectures.

## Early quality and cost controls

Once you have normalized all token streams from various providers into a centralized data model, you can set up automated checks. This helps identify software bugs, anomalous behavior, or unexpected cost spikes at an early stage.

### 1. Detection of unexplained spikes per model

By continuously monitoring the average token consumption per request per function (`calling_function`), you can immediately trigger alerts on anomalies. If a specific function suddenly consumes twice as many input tokens after a software release, this often points to a bug in prompt construction or the unintentional inclusion of redundant chat history.

### 2. Determining cache efficiency

A sudden drop in the percentage of `raw_cached_input_tokens` relative to the total input volume is a critical signal. This typically indicates that prompt caching is no longer working optimally. Potential causes include system instructions containing dynamic elements (such as a changing timestamp at the top of the prompt), which invalidates the cache's unique hash and causes the provider to process every call as an entirely new prompt.

### 3. Verification of the provider invoice

At the end of the month, you can compare the aggregated totals from your internal data model against the consolidated invoice and usage reports from the API provider. Minor discrepancies are normal due to rounding differences or occasional network errors. However, discrepancies greater than a fraction of a percent indicate structural issues, such as unrecorded streaming interruptions, improperly applied discount tiers, or missing error handling in your own application code.

For a broader overview of setting up alerting and threshold values, refer to the guide on [monitoring costs](https://api.llmnet.nl/en/kosten-monitoren) on the platform. Once the baseline data is reliably captured, it also becomes possible to draw comparisons with the performance metrics from the [cost-per-task benchmark](https://benchmark.llmnet.nl/en/kosten-per-taak).

## Read also

- [Monitoring LLM Infrastructure Costs](https://api.llmnet.nl/en/kosten-monitoren)

- [Allocating Costs per User in Multi-Tenant Systems](https://api.llmnet.nl/en/kosten-per-gebruiker-toerekenen)

- [Observability and Logging for AI Applications](https://api.llmnet.nl/en/observability-en-logging)

- [Per-Token Pricing Models Explained](https://hub.llmnet.nl/en/prijsmodellen-per-token-uitgelegd)

- [Tokenization Explained: From Text to Vector](https://leren.llmnet.nl/en/tokenisatie-uitgelegd)

- [Cost-Per-Task Benchmark](https://benchmark.llmnet.nl/en/kosten-per-taak)

llmnet.nl - LLM Aggregation and API Integration