Cost per token explained: why AI tokens get cheaper while GPUs cost more

Part of our AI models guide: 11 posts on choosing and running AI models.
An 8-GPU B300 server costs about $550,000 in the US, nearly three times what an H100 server cost. Yet the price of an AI answer of the same quality has fallen roughly 5 to 13 times a year since 2023. Both are true because newer GPUs write far more tokens: in NVIDIA's MLPerf v5.0 results, an 8-GPU B200 server made about three times the tokens of an H200 server, partly thanks to 4-bit maths. A token, the smallest unit of text an AI model reads and writes, is about three quarters of an English word, and AI services charge per million of them. In November 2026, list prices for the newest Claude, GPT-6 and Gemini 4 models run from $0.10 to $8 per million input tokens, and output, including hidden reasoning, costs five times input. Cheaper tokens go into longer prompts and AI agents, so total GPU spending keeps rising.
Tokens got cheaper, GPUs got dearer
By Epoch AI's count, GPT-4-level ability got about 208 times cheaper per token in under two years, as Gemini 2.0 Flash beat GPT-4 on the MMLU knowledge test. Hardware went the other way.
What | Earlier | Later |
|---|---|---|
Price per million tokens, input and output combined, for GPT-4-level ability (Epoch AI) | $37.50 (GPT-4, March 2023) | $0.18 (Gemini 2.0 Flash, February 2025) |
Purchase price per GPU in an 8-GPU server | About $24,000 (H100 server at about $190,000, SemiAnalysis, 2025) | About $69,000 (B300 server at about $550,000 in the US, Reuters, April 2026) |
What is a token?
A token is the smallest unit of text an AI model reads and writes. Before the model sees your question, a tool called a tokenizer cuts the text into these pieces. A short, common word is usually one token; a long or rare word is split into parts, as when one tokenizer turns "transformer" into "transform" and "er". Spaces, punctuation and numbers become tokens too.
Think of tokens as the units on an electricity meter. The model counts every token it reads, your question plus any documents and chat history sent with it, and every token it writes back. The bill is the units used times the price per unit, which is why AI prices are quoted per million tokens.
A rough guide for English:
- OpenAI: one English token is about four characters, or three quarters of a word, so a million tokens is about 750,000 words.
- Google: for Gemini, 100 tokens is about 60 to 80 English words.
- DeepSeek: one English character is about 0.3 tokens, one Chinese character about 0.6.
So a one-page email of about 500 English words is roughly 670 tokens. Other languages often need more tokens for the same meaning, as the language section below shows.
Images and audio become tokens too: on Gemini, a small image is 258 tokens and audio 32 tokens a second.
How tokens are billed
Input and output are priced separately. Output costs more because the model reads your prompt in one pass but writes its answer one token at a time.
What you pay for | What it is | How it is priced (list prices, 19 November 2026) |
|---|---|---|
Input | Everything you send: instructions, history, documents | $4 per million on Claude Opus 5.5, $8 on GPT-6.2 Astra, $4 on Gemini 4 Carbon from 1 December 2026 |
Output | What the model writes back | Five times input on Claude, OpenAI's GPT-6 models and Gemini 4 Carbon; three times on DeepSeek V4-Pro ($1.32 in, $3.96 out at peak hours) |
Reasoning | Hidden working-out before the answer | Billed as output by Anthropic and OpenAI; included in Gemini's output price |
Cached input | Repeated prompt text the provider has stored | 2.5% to 10% of input on Claude (5% on Opus 5.5), after a first write at 1.25 to 2 times input; 5% on GPT-6.2 Astra |
Batch | Jobs that can wait | Half price at Anthropic and Google; stacks with caching at Anthropic |
Long prompts | Very large inputs | Can cost more: Claude Haiku 5.5 is five times dearer above 100,000 tokens; above 272,000 input tokens, input on OpenAI's GPT-6 models doubles and output rises 1.5 times |
An illustration: a support bot sends 6,000 tokens in, about the average prompt in late 2025 on OpenRouter (one API for more than 600 AI models, by its own count). It gets 400 back (our assumption). On Claude Opus 5.5 that is 3.2 cents an answer, or $32,000 per million answers. Read 5,000 of those tokens, a fixed system prompt, from cache and it falls to about 1.3 cents; batch processing halves that again. Reasoning pushes the other way: add 2,000 tokens of thinking, billed as output by Anthropic, OpenAI and Google, and the uncached answer costs 7.2 cents.
How GPUs are measured in tokens
A GPU's work is measured in tokens per second; the terms below follow NVIDIA's definitions. The examples come from MLPerf, the industry benchmark run by MLCommons, on Llama 2 70B, a 2023 Meta model. Data-centre GPUs usually come eight to a server, often called a node.
Measure | Plain meaning | Example |
|---|---|---|
Time to first token | Wait before the answer starts | 450 milliseconds at most in MLPerf's interactive Llama 2 70B test |
Time per output token (inter-token latency) | Gap between new tokens | 40 milliseconds at most, about 25 tokens a second per user |
Throughput | All tokens one GPU writes per second, across all users | About 12,300 per B200 and 4,100 per H200 (NVIDIA's MLPerf v5.0 server results for 8-GPU servers, divided by 8) |
Concurrency | Users served at once | More users raise throughput until the GPU is full, but each user slows down |
That last trade-off sets your cost per token. The throughput figures come from MLPerf's looser server test (up to 2 seconds to the first token, 200 milliseconds per token), so expect fewer tokens per GPU at interactive speeds. Long prompts and agent histories also fill GPU memory; see our GPU memory guide.
Cost per million tokens on an H200 vs a B200
Our GPU cost per hour guide explains why hourly quotes differ. To turn any of them into a cost per token:
Cost per million tokens = price per hour ÷ (tokens per second × 3,600) × 1,000,000
The prices are round illustrations, not our prices and not a quote: $3.00 per GPU-hour for the H200, $4.00 for the B200.
Line | H200 server (8 GPUs) | B200 server (8 GPUs) |
|---|---|---|
Throughput, Llama 2 70B, NVIDIA's MLPerf v5.0 server results | 33,072 tokens a second | 98,443 tokens a second |
Tokens per hour | 119.1 million | 354.4 million |
Illustrative price per server-hour | $24.00 ($3.00 per GPU) | $32.00 ($4.00 per GPU) |
Cost per million tokens, busy all the time | $0.20 | $0.09 |
Busy half the time | $0.40 | $0.18 |
Busy a quarter of the time | $0.81 | $0.36 |
Against the H200 | 1.00 | A third more per hour, 55% less per token (0.45) |
Two lessons. The B200 costs more per hour but less per token, because it writes about three times the tokens. On these figures it stays cheaper per token unless its hourly price is nearly three times the H200's; our Hopper vs Blackwell guide explains the jump. And use matters as much as the chip: a busy H200 ($0.20) beats a B200 busy a quarter of the time ($0.36).
What the estimate assumes:
- These are NVIDIA's own submissions, reviewed by MLCommons, on tuned systems. The B200 ran in FP4, a 4-bit number format (FP8 uses 8 bits) that lets the chip do more work per second, so part of the gain is the format.
- Llama 2 70B is an older dense model: the whole model works on every token. Gaps differ on newer models.
- MLPerf counts the tokens the model writes, so compare with output-token prices.
- Staff, storage, networking, software and tax are left out.
Whether this beats paying per token at your volume is in our main guide. Tell us the model, prompt length and the speed your users need, and we will size the node and get you quotes from verified operators: ask for a quote. See H200 pricing and B200 pricing.
Why token counts differ by model and language
Counts are not comparable across model families. Anthropic says the tokenizer in Claude 4.7 and later makes about 30% more tokens for the same text than earlier Claude models. OpenRouter says one provider's token is not directly comparable to another's.
Language matters more, because many tokenizers learned mostly from English. Below: total tokens for the same translated sentences (not tokens per word), English = 1.00, in our arithmetic from Petrov et al.'s published counts (NeurIPS 2023).
Language | GPT-2 tokenizer (2019) | GPT-4 tokenizer (2023) | Qwen tokenizer (2023) | BLOOM tokenizer (2022, multilingual) |
|---|---|---|---|---|
English | 1.00 | 1.00 | 1.00 | 1.00 |
Chinese (Simplified) | 3.21 | 1.91 | 1.07 | 0.95 |
Arabic | 4.40 | 3.04 | 1.63 | 1.14 |
Hindi | 7.46 | 4.79 | 4.47 | 1.28 |
Malayalam | 15.24 | 9.00 | 7.31 | 1.38 |
The tokenizer sets this "language tax", and newer tokenizers have narrowed the gap. An early August 2026 study, on different texts with machine-translated Arabic, found GPT-4o's tokenizer needs 1.44 times English for Arabic and 1.72 times for Hindi. A Microsoft blog found GPT-4o cut Malayalam counts by 79% against GPT-4.
More tokens mean a bigger bill and more GPU time per user. Our India post uses a per-word measure from a 2026 study, so its multipliers look higher; both show the same pattern. By region: Indian models, Arabic models, European models and Japanese and Korean models.
Why token prices fall, and where they rise
The venture firm a16z calls the fall in token prices "LLMflation": about 10 times a year at equal ability. It credits better GPUs, 4-bit number formats, better serving software, smaller models that match older big ones, and competition from open-weight models. Open-weight means anyone can download the weights, the trained numbers that make up the model (see open-weight vs closed models). Mixture-of-experts (MoE) models help too, waking only a small part of the model for each token.
Measure | Finding | Source, date |
|---|---|---|
Cost of a given performance level | About 47% lower each quarter since 2023, roughly 13 times a year | Epoch AI, September 2026 |
Price of a given benchmark score | 5 to 10 times cheaper a year; better algorithms alone about 3 times | Gundlach et al., March 2026 |
Average price paid per million tokens on OpenRouter | January to October 2026: closed models $1.21 to $0.52, open-weight models $0.28 to $0.084; the average also reflects model mix and caching | OpenRouter, October 2026 |
DeepSeek V4-Pro | Rose: a 75% cut in May 2026, then from 16 August peak-hour output at 4.5 times the old price and cached input up to 1,100% higher | DeepSeek; Caixin, May and August 2026 |
Gemini 4 Carbon | Rises: $2 in and $10 out at launch, $4 and $20 from 1 December 2026 | Google, November 2026 |
Top models fall more slowly: Claude Fable 5.2, at $8 in and $40 out, still costs about 43% of GPT-4's 2023 combined price. For the next GPU generation, see our Vera Rubin guide.
Why each new GPU costs more
Each generation carries more high-bandwidth memory (HBM), the fast memory that holds the model's weights. Per GPU: 80 GB on the H100, 141 GB on the H200, 180 GB on the B200 and 288 GB on the B300.
Epoch AI estimates a B200 costs about $6,400 to make, nearly half of it ($2,900) memory. Jensen Huang, NVIDIA's chief executive, put the B200 at $30,000 to $40,000 in March 2024; NVIDIA later said pricing depends on the system. Epoch reads that as about 80% chip-level gross margin, likely lower in practice.
Systems are growing too, from 8-GPU servers (NVIDIA's HGX design) to 72-GPU NVL72 racks (see GPU form factors and NVL72 racks explained). SemiAnalysis puts a GB200 NVL72's total cost of ownership at about 1.6 times an H100's, per GPU. Yet newer GPUs write so many more tokens that cost per token still falls, as the worked example shows. Rental rates have risen this year too, as our GPU cost per hour guide shows.
Our own "from" prices climb the same ladder, from $2.95 per GPU-hour for the H100 to $4.50 for the B300. These are the floor on our longest term, in US dollars before tax; a one-year term costs more. See GPU server prices; GB300 NVL72 racks are priced on request.
The Jevons paradox: why GPU spending grows
The Jevons paradox says that when using a resource gets cheaper, people use so much more of it that total spending rises.
Signal | Figure | Source, date |
|---|---|---|
Average prompt per request on OpenRouter | About 1,500 to over 6,000 tokens, early 2024 to late 2025 | OpenRouter and a16z, January 2026 |
Share of OpenRouter tokens from AI agents | 8% in October 2025, 76% in October 2026 | OpenRouter, October 2026 |
Tokens per task | One agent about 4 times a chat; multi-agent systems about 15 times | Anthropic, June 2025 |
Cost of running frontier models | Rising about 3 to 18 times a year | Gundlach et al., March 2026 |
Price is not the main driver: the OpenRouter and a16z study found a 10% price cut brought only 0.5% to 0.7% more use. Growth comes from new work such as agents and reasoning, and far more tokens means more GPU-hours.
What this means for buyers
- Compare the cost of a finished task, not the price per million tokens.
- Ask for tokens per second on your model, prompt length and languages, at your users' speed (see our questions to ask a GPU provider).
- Plan for use: idle hours raise the cost of every token, and agents and long prompts need memory as well as speed.
- Prototype on per-token APIs and move steady volume to dedicated servers when the numbers work (see pay per token, rent or buy).
From cost per token to a server quote
Once you have a cost per million tokens for your workload, the missing input is a real server price to put into the same formula. We supply that: quotes for 8-GPU H200 and B200 servers like those above, as well as H100 and B300, from verified operators in North America, the UK, Europe, India, Asia Pacific and the Gulf. Each quote is stated per GPU-hour so it drops straight into the sum, but the server is contracted for a year or longer and paid in advance; we do not rent by the hour.
Request a quote you can run through the formula, or see which servers we source.
Where these figures come from
Pages checked on 19 November 2026 unless a date is given; prices change often.
- Tokens: OpenAI; Google; DeepSeek; Hugging Face; NVIDIA (17 March 2025).
- Prices, caching and tokenizers: Anthropic pricing and thinking guide; OpenAI pricing and reasoning guide; Gemini; DeepSeek; OpenRouter docs.
- Languages: Petrov et al., NeurIPS 2023 (ratios are our arithmetic); Roy, Roy and Patel, preprint (10 August 2026); Microsoft Azure AI blog (20 May 2024).
- Throughput: NVIDIA, MLPerf Inference v5.0 (2 April 2025); MLCommons inference rules and Llama 2 70B reference code.
- Price trends: Epoch AI, LLM inference price trends (12 March 2025) and The plunging price of thought (22 September 2026); a16z, LLMflation (12 November 2024); Gundlach et al. (March 2026); DeepSeek price changes: Caixin, 25 May 2026 and 14 August 2026.
- OpenRouter usage: October 2026 data; About page; State of AI study with a16z (January 2026).
- Hardware: SemiAnalysis (20 August 2025); Reuters, via Yahoo Finance (30 April 2026); Epoch AI, B200 cost (10 December 2025); CNBC, via The Star (19 March 2024).
- Agents: Anthropic, multi-agent research system (June 2025).