LLM inference providers explained: how tokens are sold and when you need GPUs

Part of our AI models guide: 11 posts on choosing and running AI models.
On 19 November 2026, 37 companies sold the same AI model, GLM-5.3, through OpenRouter, one API (a web service your software calls) for hundreds of models, and the dearest charged five times the cheapest per output token. A token is the smallest unit of text an AI model reads and writes, about three quarters of an English word. Only one of the 37, Z.ai, made GLM-5.3. Most of the rest are inference providers: they rent or buy GPU servers, run open models and sell the output by the million tokens, as a bakery buys ovens and sells bread by the loaf. Their profit turns on how many tokens each server handles an hour and how busy it stays. For most businesses, an API is the simplest and usually the cheapest way in: no GPUs, no software to patch and nobody on call at 3 a.m.
What an inference provider does
An LLM inference provider, also called an LLM API provider, runs AI models on GPU servers and sells the answers through an API, priced per million tokens read (input) and written (output). Inference means running a trained model; training is building it. Three kinds of seller offer this inference as a service:
- Model makers: OpenAI, Anthropic, Google, DeepSeek and Mistral sell their own models.
- Hyperscalers: Amazon Bedrock, Microsoft Foundry (formerly Azure AI Foundry) and Google's Gemini Enterprise Agent Platform (formerly Vertex AI) sell many makers' models.
- Independent inference providers: they serve open-weight models, whose files anyone can download (see open-weight vs closed models).
Independent providers sell through their own APIs and through aggregators such as OpenRouter, which send each request to one of many sellers and switch if one fails.
Which models sell most
On OpenRouter, developers send the most tokens to cheap Chinese open models: in the week to 18 November 2026, 8 of its 10 most-used models came from Chinese labs, led by DeepSeek V4.1 Flash 1017. The reason is price: AI agents, software that calls a model over and over, sent 76% of October's tokens. Tokens are not money, though: closed models such as Claude and GPT took about 75% of October's paid spend. Who leads on users, spend and downloads, and why: the most popular AI models.
How tokens are sold: serverless or dedicated
Serverless means you pay per token and never see a server, so the seller carries idle time. Dedicated means you reserve capacity and pay by the hour, used or not: Microsoft bills Foundry's provisioned throughput units (PTUs) hourly, they cannot be paused, and even a reservation does not guarantee capacity.
Option | How you pay | Who pays for idle time | Examples |
|---|---|---|---|
Serverless, standard | Per million tokens | Seller | Makers' APIs; Bedrock on-demand; Foundry Standard; Agent Platform Standard pay-as-you-go |
Batch or flex | About half price, slower | Seller | Bedrock Batch and Flex, OpenAI Batch, Agent Platform Flex and Batch |
Priority or fast | A premium per token | Seller | Bedrock Priority (75% more); OpenAI Fast (double) |
Dedicated or provisioned | Per hour, used or not | Buyer | Bedrock Provisioned Throughput (no commitment, or 1 or 6 months); Foundry PTUs (reserved for a month or a year); Agent Platform (1 week to 1 year, a year 26% cheaper than a month) |
Own servers, rented on a term or bought | Per hour or upfront, used or not | Buyer |
An inference provider renting whole servers on a term is in the buyer's seat: it pays for every hour and must fill them.
What an API saves a business
Most businesses want answers, not GPUs:
Job | If you run the model yourself | Through an API |
|---|---|---|
Idle hours | You pay for the servers every hour, busy or not | Per-token pricing: only the tokens you use |
Hardware | Buy or rent servers; one NVIDIA DGX B200 draws up to about 14.3 kW, needing data-centre power and cooling | None |
Serving software | Install, tune and upgrade it: the vLLM serving engine shipped 21 minor versions in 2026 | The provider's job |
Security | A critical vLLM flaw, rated 9.8 out of 10 for severity, in February 2026, and 40 security advisories from 6 October to 18 November | The provider's job |
On-call | Google's SRE book puts a single-site 24/7 rota at eight engineers minimum | The provider's job |
New models | Download, test and re-tune each one | Change one name in your code |
Data protection | Your own controls and audits | Bedrock: makers cannot see prompts; India-only processing for some OpenAI and Claude models. Foundry: the same for models sold by Azure. Google: no training on your data without permission |
APIs churn too. Anthropic gives at least 60 days' notice before retiring a model, OpenAI at least six months for generally available ones, and OpenAI is closing self-serve fine-tuning (further training on your own data): new customers already cannot use it, and its remaining existing customers cannot start new jobs from 6 January 2027. Keeping processing in one region costs about 10% more at OpenAI (for models released from 5 March 2026) and at Anthropic (for US-only processing).
Among open-weight users in production, ETR (September 2026, via Techstrong.ai) found 46% self-manage in the cloud, 42% run on premises, 29% use hyperscalers and 27% managed inference providers; many use several.
How one GPU serves many customers
Serving engines such as vLLM, SGLang, TensorRT-LLM and NVIDIA Dynamo keep each GPU busy with many customers at once:
Technique | Plain meaning | Why it cuts cost |
|---|---|---|
Continuous batching | New requests join the running batch at every step, like a bus that stops everywhere | Many customers share one GPU |
Paged KV cache | The KV cache is the model's notes on what it has read, stored in small pages | Older systems wasted 60% to 80% of this memory, vLLM's PagedAttention under 4%, so more requests fit |
Prefix caching | Repeated text, such as a fixed system prompt, is reused, not read again | Why cached input is cheap |
Splitting prefill and decode | Reading the prompt needs compute; writing the answer needs memory speed (NVIDIA) | Each runs on GPUs tuned for it |
Quantisation (FP8, FP4) | The model's numbers stored in 8 or 4 bits, not 16 | More tokens per GPU; NVIDIA reports NVFP4 is about 1.8 times smaller than FP8, with 1% or less accuracy loss on DeepSeek-R1-0528 |
The gains stack: one GPU can handle tens of thousands of tokens a second, reading and writing together.
The unit economics, worked through
Apply the formula from our cost per token guide to one 8-GPU NVIDIA B200 server selling OpenAI's open-weight gpt-oss-120b.
One 8-GPU B200 server selling gpt-oss-120b (illustration) | Figure |
|---|---|
Throughput per GPU at 50 tokens a second per user (InferenceX) | 44,489 tokens a second, input plus output |
Tokens per server-hour | About 1.28 billion (1.14 billion in, 142 million out) |
Server price per hour | $32 ($4.00 per GPU-hour, an illustration, not our price) |
Cost per million tokens if busy all the time, half the time, a quarter of the time | $0.025, $0.05, $0.10 |
Revenue per fully busy hour: cheapest, median and one of the dearest OpenRouter listings | About $56, $185, $505 |
Share of the day it must be busy to cover $32: cheapest, median, dearest | About 58%, 17%, 6% |
Speed per user | Tokens a second per GPU, in plus out | Busy share needed: cheapest listing | Median listing |
|---|---|---|---|
50 tokens a second | 44,489 | About 58% | About 17% |
100 tokens a second | 29,548 | About 87% | About 26% |
150 tokens a second | 20,866 | Never (would need 123%) | About 37% |
- Throughput: SemiAnalysis's InferenceX run of 31 May 2026 (B200, 4-bit, vLLM), counting input plus output tokens. Our tokens guide's MLPerf figures count only output, so the two costs per million are not comparable.
- Each request: 8,000 tokens in, 1,000 out, random inputs, no caching; revenue scales with load.
- OpenRouter prices for gpt-oss-120b per million tokens, input then output: cheapest $0.03 and $0.15, median $0.10 and $0.50, one of the dearest $0.35 and $0.75.
- $4.00 per GPU-hour is a round illustration, not our price or a quote.
- Staff, network, storage, software and platform fees are left out.
At full load, $32 of server time handles tokens that sell for $56 to $505. At the cheapest price, though, the server must be busy over half the day. And speed costs throughput: 50 tokens a second is already several times reading speed, and faster service per user means fewer users per GPU, which is our reading of why fast or priority tiers cost more.
Do inference providers make money?
Yes, when their GPUs stay busy. Published checks say the margin on a busy GPU is large:
Source, date | What was measured | Result | Catch |
|---|---|---|---|
DeepSeek, 1 March 2025 | 24 hours of its own V3 and R1 serving on H800 GPUs, costed at $2 per GPU-hour | $87,072 of cost against $562,027 if all tokens were billed at R1 prices: a "cost profit margin" of 545% (profit 5.45 times cost) | Real revenue is much lower, DeepSeek says: free web and app use, cheaper V3, night discounts |
InferenceX (SemiAnalysis), November 2026 | Makers' own API prices against self-hosted cost on an AI-agent workload, GPUs costed as if owned at hyperscaler volume | GLM-5.3: API $0.303, self-hosted on B300 $0.030 per million tokens (10 times less). Kimi K3.1 Pro: 10 times less on B300, 1.8 times on B200 (different serving software on each) | Mostly cached input |
Why the same model costs five times more
The pattern repeats across four popular open models (OpenRouter listings on 19 November 2026; counts include every tier and, for GLM, DeepSeek and Kimi, the maker's own listing):
Model | Listings (sellers) | Output price per million tokens | Median | Precision shown |
|---|---|---|---|---|
GLM-5.3 | 49 (37) | $1.32 to $6.60 | $4.40 | 18 fp4, 4 nvfp4, 12 fp8, 15 not stated |
DeepSeek V4.1 Flash (1017) | 50 (41) | $0.15 to $2.40 | $0.69 | 23 fp8, 8 fp4 or mxfp4, 19 not stated |
Kimi K3.1 Pro | 27 (20) | $6 to $22 | $10 | 17 fp4 or mxfp4, 1 fp8, 1 bf16, 8 not stated |
gpt-oss-120b | 24 (21) | $0.15 to $0.95 | $0.50 | 10 fp4, 1 int4, 2 fp8, 4 bf16 or fp16, 7 not stated |
- Routing: OpenRouter shows each provider's own price and charges a fee when you buy credits. Its default routing weighs price after cache hits, speed, time to first token, rate limits, reliability and quality, so the cheapest seller does not take all the traffic.
- Precision: fp4, nvfp4, mxfp4 and int4 are 4-bit formats, fp8 is 8-bit, and bf16 and fp16 are 16-bit; fewer bits mean more tokens per GPU. The cheapest gpt-oss-120b listings are 4-bit, but some 16-bit ones cost almost as little.
- Quality: the same weights can give different results: OpenRouter launched Exacto, endpoints chosen for accurate tool use, and Kimi's maker, Moonshot AI, published a vendor verifier after results differed across providers.
The risks of selling tokens
- Price wars: OpenRouter's average price paid for open-weight models fell 70%, from $0.28 to $0.084 per million tokens, between January and October 2026. Since May 2026, monthly tokens for GLM's maker Z.ai grew about 20 times, but spend only about 6 times.
- Model churn: four GLM-5.x versions reached OpenRouter between 11 February and 18 August 2026. Software updates keep raising output after launch: DeepSeek V4 Pro on B200 gained about 1.7 times in 43 days (SemiAnalysis), and about 5 times on GB300 from April to June 2026 (SGLang). Providers that re-tune fast win; each new version restarts the work.
- Idle capacity: DeepSeek serves on all its servers at the daytime peak and moves some to research and training at night; its inference fleet averaged about 82% of peak.
- Licences: some open-weight licences need a separate agreement to sell the model as a service, such as Kimi K3.1 Pro once the seller passes $20 million in revenue in any 12 months, and Mistral Large 4 at any revenue; internal use is exempt (not legal advice).
When owning or renting GPUs wins
The buyer's break-even is the seller's, seen from the other side. OpenRouter's formula: GPU rent per hour ÷ API cost of one fully busy GPU-hour of tokens = the share of the day the GPUs must be busy. Our gpt-oss-120b server needs about 17% at the median API price and 58% at the cheapest, before staff. OpenRouter's own examples, which leave out engineering time, are tougher: DeepSeek V4.1 Flash on four H200s never broke even against the cheapest endpoint that keeps no data; on two B200s it needed 37% to 76% at the lower rent it tested, and at the higher rent it never matched the cheapest.
Self-hosting tends to win for:
- steady, high volume of a large open model;
- data that must stay on hardware you control (zero data retention, a provider's promise not to store prompts, is not that: prompts still run on its GPUs, OpenRouter notes);
- models fine-tuned on your own data;
- freezing a model version;
- response times or a service guarantee you set yourself.
In India, the Reserve Bank of India requires payment-system data to be stored only in India, and SEBI, the markets regulator, requires regulated firms' cloud data to stay there; the DPDP Act, India's data protection law, has no blanket localisation rule. Not legal advice; self-hosting alone makes no one compliant.
API prices by model are in our sizing guide.
Selling tokens? Start with the server cost
Our side of the formula is the server: whole 8-GPU NVIDIA H100, H200, B200 or B300 machines, or GB300 NVL72 racks on request, from verified operators in North America, the UK, Europe, India, Asia Pacific and the Gulf, for at least one year, billed upfront. Quotes are per GPU-hour for comparison; we do not rent by the hour or sell tokens. Tell us the model, traffic and per-user speed you sell or need, and any data rules you must meet, and we will size the server and get quotes: ask for a quote, or see the servers we source. If an API already covers your volume and data rules, stay on it.
Where these figures come from
OpenRouter figures were pulled on 19 November 2026 and change daily. Tokens per hour, revenue and break-even shares are our arithmetic.
- Serving: vLLM (6 November 2026) and PagedAttention (20 June 2023); NVIDIA on Dynamo (18 March 2025) and NVFP4 (24 June 2025).
- InferenceX (SemiAnalysis): gpt-oss-120b on B200 (run of 31 May 2026) and agentic costs, viewed 19 November 2026.
- OpenRouter: providers for GLM-5.3, DeepSeek V4.1 Flash, Kimi K3.1 Pro and gpt-oss-120b; models list; rankings (week to 18 November 2026); routing; tiers; Exacto; October 2026 data; self-hosted vs API (12 November 2026). Kimi Vendor Verifier.
- Margins: DeepSeek (1 March 2025). Software gains: SemiAnalysis (9 June 2026); PyTorch (23 June 2026).
- Ways to buy: Bedrock pricing; Microsoft PTU billing; Google consumption options; OpenAI pricing.
- Licences and hardware: Kimi K3.1 Pro licence; NVIDIA DGX B200; vLLM CVE-2026-22778 and advisories; Google SRE book.
- Data and lifecycle: Bedrock data protection and FAQs; Bedrock India for OpenAI (18 August 2026) and Claude (29 September 2026); Foundry privacy; Google data governance; deprecations at Anthropic and OpenAI; residency at OpenAI and Anthropic.
- Surveys and rules: ETR via Techstrong.ai (22 September 2026); RBI payment data (2018); SEBI cloud framework (6 March 2023).