Skip to content

Cost per token explained: why AI tokens get cheaper while GPUs cost more

19 November 2026Vinod Chacko
Cost per token explained: why AI tokens get cheaper while GPUs cost more

Part of our AI models guide: 11 posts on choosing and running AI models.

An 8-GPU B300 server costs about $550,000 in the US, nearly three times what an H100 server cost. Yet the price of an AI answer of the same quality has fallen roughly 5 to 13 times a year since 2023. Both are true because newer GPUs write far more tokens: in NVIDIA's MLPerf v5.0 results, an 8-GPU B200 server made about three times the tokens of an H200 server, partly thanks to 4-bit maths. A token, the smallest unit of text an AI model reads and writes, is about three quarters of an English word, and AI services charge per million of them. In November 2026, list prices for the newest Claude, GPT-6 and Gemini 4 models run from $0.10 to $8 per million input tokens, and output, including hidden reasoning, costs five times input. Cheaper tokens go into longer prompts and AI agents, so total GPU spending keeps rising.

Tokens got cheaper, GPUs got dearer

By Epoch AI's count, GPT-4-level ability got about 208 times cheaper per token in under two years, as Gemini 2.0 Flash beat GPT-4 on the MMLU knowledge test. Hardware went the other way.

What

Earlier

Later

Price per million tokens, input and output combined, for GPT-4-level ability (Epoch AI)

$37.50 (GPT-4, March 2023)

$0.18 (Gemini 2.0 Flash, February 2025)

Purchase price per GPU in an 8-GPU server

About $24,000 (H100 server at about $190,000, SemiAnalysis, 2025)

About $69,000 (B300 server at about $550,000 in the US, Reuters, April 2026)

What is a token?

A token is the smallest unit of text an AI model reads and writes. Before the model sees your question, a tool called a tokenizer cuts the text into these pieces. A short, common word is usually one token; a long or rare word is split into parts, as when one tokenizer turns "transformer" into "transform" and "er". Spaces, punctuation and numbers become tokens too.

Think of tokens as the units on an electricity meter. The model counts every token it reads, your question plus any documents and chat history sent with it, and every token it writes back. The bill is the units used times the price per unit, which is why AI prices are quoted per million tokens.

A rough guide for English:

  • OpenAI: one English token is about four characters, or three quarters of a word, so a million tokens is about 750,000 words.
  • Google: for Gemini, 100 tokens is about 60 to 80 English words.
  • DeepSeek: one English character is about 0.3 tokens, one Chinese character about 0.6.

So a one-page email of about 500 English words is roughly 670 tokens. Other languages often need more tokens for the same meaning, as the language section below shows.

Images and audio become tokens too: on Gemini, a small image is 258 tokens and audio 32 tokens a second.

How tokens are billed

Input and output are priced separately. Output costs more because the model reads your prompt in one pass but writes its answer one token at a time.

What you pay for

What it is

How it is priced (list prices, 19 November 2026)

Input

Everything you send: instructions, history, documents

$4 per million on Claude Opus 5.5, $8 on GPT-6.2 Astra, $4 on Gemini 4 Carbon from 1 December 2026

Output

What the model writes back

Five times input on Claude, OpenAI's GPT-6 models and Gemini 4 Carbon; three times on DeepSeek V4-Pro ($1.32 in, $3.96 out at peak hours)

Reasoning

Hidden working-out before the answer

Billed as output by Anthropic and OpenAI; included in Gemini's output price

Cached input

Repeated prompt text the provider has stored

2.5% to 10% of input on Claude (5% on Opus 5.5), after a first write at 1.25 to 2 times input; 5% on GPT-6.2 Astra

Batch

Jobs that can wait

Half price at Anthropic and Google; stacks with caching at Anthropic

Long prompts

Very large inputs

Can cost more: Claude Haiku 5.5 is five times dearer above 100,000 tokens; above 272,000 input tokens, input on OpenAI's GPT-6 models doubles and output rises 1.5 times

An illustration: a support bot sends 6,000 tokens in, about the average prompt in late 2025 on OpenRouter (one API for more than 600 AI models, by its own count). It gets 400 back (our assumption). On Claude Opus 5.5 that is 3.2 cents an answer, or $32,000 per million answers. Read 5,000 of those tokens, a fixed system prompt, from cache and it falls to about 1.3 cents; batch processing halves that again. Reasoning pushes the other way: add 2,000 tokens of thinking, billed as output by Anthropic, OpenAI and Google, and the uncached answer costs 7.2 cents.

How GPUs are measured in tokens

A GPU's work is measured in tokens per second; the terms below follow NVIDIA's definitions. The examples come from MLPerf, the industry benchmark run by MLCommons, on Llama 2 70B, a 2023 Meta model. Data-centre GPUs usually come eight to a server, often called a node.

Measure

Plain meaning

Example

Time to first token

Wait before the answer starts

450 milliseconds at most in MLPerf's interactive Llama 2 70B test

Time per output token (inter-token latency)

Gap between new tokens

40 milliseconds at most, about 25 tokens a second per user

Throughput

All tokens one GPU writes per second, across all users

About 12,300 per B200 and 4,100 per H200 (NVIDIA's MLPerf v5.0 server results for 8-GPU servers, divided by 8)

Concurrency

Users served at once

More users raise throughput until the GPU is full, but each user slows down

That last trade-off sets your cost per token. The throughput figures come from MLPerf's looser server test (up to 2 seconds to the first token, 200 milliseconds per token), so expect fewer tokens per GPU at interactive speeds. Long prompts and agent histories also fill GPU memory; see our GPU memory guide.

Cost per million tokens on an H200 vs a B200

Our GPU cost per hour guide explains why hourly quotes differ. To turn any of them into a cost per token:

Cost per million tokens = price per hour ÷ (tokens per second × 3,600) × 1,000,000

The prices are round illustrations, not our prices and not a quote: $3.00 per GPU-hour for the H200, $4.00 for the B200.

Line

H200 server (8 GPUs)

B200 server (8 GPUs)

Throughput, Llama 2 70B, NVIDIA's MLPerf v5.0 server results

33,072 tokens a second

98,443 tokens a second

Tokens per hour

119.1 million

354.4 million

Illustrative price per server-hour

$24.00 ($3.00 per GPU)

$32.00 ($4.00 per GPU)

Cost per million tokens, busy all the time

$0.20

$0.09

Busy half the time

$0.40

$0.18

Busy a quarter of the time

$0.81

$0.36

Against the H200

1.00

A third more per hour, 55% less per token (0.45)

Two lessons. The B200 costs more per hour but less per token, because it writes about three times the tokens. On these figures it stays cheaper per token unless its hourly price is nearly three times the H200's; our Hopper vs Blackwell guide explains the jump. And use matters as much as the chip: a busy H200 ($0.20) beats a B200 busy a quarter of the time ($0.36).

What the estimate assumes:

  • These are NVIDIA's own submissions, reviewed by MLCommons, on tuned systems. The B200 ran in FP4, a 4-bit number format (FP8 uses 8 bits) that lets the chip do more work per second, so part of the gain is the format.
  • Llama 2 70B is an older dense model: the whole model works on every token. Gaps differ on newer models.
  • MLPerf counts the tokens the model writes, so compare with output-token prices.
  • Staff, storage, networking, software and tax are left out.

Whether this beats paying per token at your volume is in our main guide. Tell us the model, prompt length and the speed your users need, and we will size the node and get you quotes from verified operators: ask for a quote. See H200 pricing and B200 pricing.

Why token counts differ by model and language

Counts are not comparable across model families. Anthropic says the tokenizer in Claude 4.7 and later makes about 30% more tokens for the same text than earlier Claude models. OpenRouter says one provider's token is not directly comparable to another's.

Language matters more, because many tokenizers learned mostly from English. Below: total tokens for the same translated sentences (not tokens per word), English = 1.00, in our arithmetic from Petrov et al.'s published counts (NeurIPS 2023).

Language

GPT-2 tokenizer (2019)

GPT-4 tokenizer (2023)

Qwen tokenizer (2023)

BLOOM tokenizer (2022, multilingual)

English

1.00

1.00

1.00

1.00

Chinese (Simplified)

3.21

1.91

1.07

0.95

Arabic

4.40

3.04

1.63

1.14

Hindi

7.46

4.79

4.47

1.28

Malayalam

15.24

9.00

7.31

1.38

The tokenizer sets this "language tax", and newer tokenizers have narrowed the gap. An early August 2026 study, on different texts with machine-translated Arabic, found GPT-4o's tokenizer needs 1.44 times English for Arabic and 1.72 times for Hindi. A Microsoft blog found GPT-4o cut Malayalam counts by 79% against GPT-4.

More tokens mean a bigger bill and more GPU time per user. Our India post uses a per-word measure from a 2026 study, so its multipliers look higher; both show the same pattern. By region: Indian models, Arabic models, European models and Japanese and Korean models.

Why token prices fall, and where they rise

The venture firm a16z calls the fall in token prices "LLMflation": about 10 times a year at equal ability. It credits better GPUs, 4-bit number formats, better serving software, smaller models that match older big ones, and competition from open-weight models. Open-weight means anyone can download the weights, the trained numbers that make up the model (see open-weight vs closed models). Mixture-of-experts (MoE) models help too, waking only a small part of the model for each token.

Measure

Finding

Source, date

Cost of a given performance level

About 47% lower each quarter since 2023, roughly 13 times a year

Epoch AI, September 2026

Price of a given benchmark score

5 to 10 times cheaper a year; better algorithms alone about 3 times

Gundlach et al., March 2026

Average price paid per million tokens on OpenRouter

January to October 2026: closed models $1.21 to $0.52, open-weight models $0.28 to $0.084; the average also reflects model mix and caching

OpenRouter, October 2026

DeepSeek V4-Pro

Rose: a 75% cut in May 2026, then from 16 August peak-hour output at 4.5 times the old price and cached input up to 1,100% higher

DeepSeek; Caixin, May and August 2026

Gemini 4 Carbon

Rises: $2 in and $10 out at launch, $4 and $20 from 1 December 2026

Google, November 2026

Top models fall more slowly: Claude Fable 5.2, at $8 in and $40 out, still costs about 43% of GPT-4's 2023 combined price. For the next GPU generation, see our Vera Rubin guide.

Why each new GPU costs more

Each generation carries more high-bandwidth memory (HBM), the fast memory that holds the model's weights. Per GPU: 80 GB on the H100, 141 GB on the H200, 180 GB on the B200 and 288 GB on the B300.

Epoch AI estimates a B200 costs about $6,400 to make, nearly half of it ($2,900) memory. Jensen Huang, NVIDIA's chief executive, put the B200 at $30,000 to $40,000 in March 2024; NVIDIA later said pricing depends on the system. Epoch reads that as about 80% chip-level gross margin, likely lower in practice.

Systems are growing too, from 8-GPU servers (NVIDIA's HGX design) to 72-GPU NVL72 racks (see GPU form factors and NVL72 racks explained). SemiAnalysis puts a GB200 NVL72's total cost of ownership at about 1.6 times an H100's, per GPU. Yet newer GPUs write so many more tokens that cost per token still falls, as the worked example shows. Rental rates have risen this year too, as our GPU cost per hour guide shows.

Our own "from" prices climb the same ladder, from $2.95 per GPU-hour for the H100 to $4.50 for the B300. These are the floor on our longest term, in US dollars before tax; a one-year term costs more. See GPU server prices; GB300 NVL72 racks are priced on request.

The Jevons paradox: why GPU spending grows

The Jevons paradox says that when using a resource gets cheaper, people use so much more of it that total spending rises.

Signal

Figure

Source, date

Average prompt per request on OpenRouter

About 1,500 to over 6,000 tokens, early 2024 to late 2025

OpenRouter and a16z, January 2026

Share of OpenRouter tokens from AI agents

8% in October 2025, 76% in October 2026

OpenRouter, October 2026

Tokens per task

One agent about 4 times a chat; multi-agent systems about 15 times

Anthropic, June 2025

Cost of running frontier models

Rising about 3 to 18 times a year

Gundlach et al., March 2026

Price is not the main driver: the OpenRouter and a16z study found a 10% price cut brought only 0.5% to 0.7% more use. Growth comes from new work such as agents and reasoning, and far more tokens means more GPU-hours.

What this means for buyers

  1. Compare the cost of a finished task, not the price per million tokens.
  2. Ask for tokens per second on your model, prompt length and languages, at your users' speed (see our questions to ask a GPU provider).
  3. Plan for use: idle hours raise the cost of every token, and agents and long prompts need memory as well as speed.
  4. Prototype on per-token APIs and move steady volume to dedicated servers when the numbers work (see pay per token, rent or buy).

From cost per token to a server quote

Once you have a cost per million tokens for your workload, the missing input is a real server price to put into the same formula. We supply that: quotes for 8-GPU H200 and B200 servers like those above, as well as H100 and B300, from verified operators in North America, the UK, Europe, India, Asia Pacific and the Gulf. Each quote is stated per GPU-hour so it drops straight into the sum, but the server is contracted for a year or longer and paid in advance; we do not rent by the hour.

Request a quote you can run through the formula, or see which servers we source.

Where these figures come from

Pages checked on 19 November 2026 unless a date is given; prices change often.

Frequently asked questions

What is a token in AI?
A token is the smallest unit of text an AI model reads and writes, often a short word or part of a longer one. AI services count tokens like units on a meter and bill per million. In English, one token is about three quarters of a word, so a million tokens is about 750,000 words, by OpenAI’s rule of thumb.
Why do output tokens cost more than input tokens?
A model reads your prompt in one pass but writes its answer one token at a time, so each output token ties up the GPU for longer. As of 19 November 2026, output costs five times input on Claude, OpenAI’s GPT-6 models and Gemini 4 Carbon, and three times on DeepSeek V4-Pro.
Do you pay for reasoning or thinking tokens?
Yes. Anthropic and OpenAI bill reasoning tokens as output even though you do not see them, and Google’s Gemini output price includes thinking tokens. OpenAI says one request can use a few hundred to tens of thousands of them.
Why does AI cost more in Hindi, Arabic or Malayalam than in English?
Many tokenizers learned mostly from English, so they cut other scripts into more pieces. On GPT-4’s tokenizer, the same translated sentences took about 3 times the tokens of English in Arabic, 4.8 times in Hindi and 9 times in Malayalam (total tokens, not per word; our arithmetic from Petrov et al., NeurIPS 2023). Newer tokenizers have narrowed the gap but not closed it.
Why are AI tokens getting cheaper while GPUs cost more?
Tokens get cheaper because of better GPUs, 4-bit number formats, better serving software, smaller models that match older large ones, and competition from downloadable open-weight models. Studies put the fall at roughly 5 to 13 times a year since 2023 at equal ability. Each new GPU costs more but writes far more tokens: in NVIDIA’s MLPerf v5.0 results, an 8-GPU B200 server made about three times the tokens of an H200 server, partly thanks to 4-bit maths.
How do I work out the cost per million tokens on a GPU?
Divide the hourly price by the tokens the GPU writes in an hour (tokens per second times 3,600), then multiply by one million. Use throughput measured on your own model at the speed your users need, and count idle hours, which raise the cost of every token.
Will cheaper tokens reduce demand for GPUs?
So far the opposite has happened. AI agents made 76% of OpenRouter’s tokens in October 2026, up from 8% a year earlier, and its average prompt grew about four times between early 2024 and late 2025. Gundlach et al. (March 2026) find the cost of running frontier models rising 3 to 18 times a year.

Need GPU capacity?

Tell us your GPU count, term and region. We source the best-fit nodes across our provider network in North America, the UK, Europe, India, Asia Pacific and the Gulf, and confirm availability and lead time for your region before you order.