Skip to content

LLM inference providers explained: how tokens are sold and when you need GPUs

19 November 2026Vinod Chacko
LLM inference providers explained: how tokens are sold and when you need GPUs

Part of our AI models guide: 11 posts on choosing and running AI models.

On 19 November 2026, 37 companies sold the same AI model, GLM-5.3, through OpenRouter, one API (a web service your software calls) for hundreds of models, and the dearest charged five times the cheapest per output token. A token is the smallest unit of text an AI model reads and writes, about three quarters of an English word. Only one of the 37, Z.ai, made GLM-5.3. Most of the rest are inference providers: they rent or buy GPU servers, run open models and sell the output by the million tokens, as a bakery buys ovens and sells bread by the loaf. Their profit turns on how many tokens each server handles an hour and how busy it stays. For most businesses, an API is the simplest and usually the cheapest way in: no GPUs, no software to patch and nobody on call at 3 a.m.

What an inference provider does

An LLM inference provider, also called an LLM API provider, runs AI models on GPU servers and sells the answers through an API, priced per million tokens read (input) and written (output). Inference means running a trained model; training is building it. Three kinds of seller offer this inference as a service:

  • Model makers: OpenAI, Anthropic, Google, DeepSeek and Mistral sell their own models.
  • Hyperscalers: Amazon Bedrock, Microsoft Foundry (formerly Azure AI Foundry) and Google's Gemini Enterprise Agent Platform (formerly Vertex AI) sell many makers' models.
  • Independent inference providers: they serve open-weight models, whose files anyone can download (see open-weight vs closed models).

Independent providers sell through their own APIs and through aggregators such as OpenRouter, which send each request to one of many sellers and switch if one fails.

Which models sell most

On OpenRouter, developers send the most tokens to cheap Chinese open models: in the week to 18 November 2026, 8 of its 10 most-used models came from Chinese labs, led by DeepSeek V4.1 Flash 1017. The reason is price: AI agents, software that calls a model over and over, sent 76% of October's tokens. Tokens are not money, though: closed models such as Claude and GPT took about 75% of October's paid spend. Who leads on users, spend and downloads, and why: the most popular AI models.

How tokens are sold: serverless or dedicated

Serverless means you pay per token and never see a server, so the seller carries idle time. Dedicated means you reserve capacity and pay by the hour, used or not: Microsoft bills Foundry's provisioned throughput units (PTUs) hourly, they cannot be paused, and even a reservation does not guarantee capacity.

Option

How you pay

Who pays for idle time

Examples

Serverless, standard

Per million tokens

Seller

Makers' APIs; Bedrock on-demand; Foundry Standard; Agent Platform Standard pay-as-you-go

Batch or flex

About half price, slower

Seller

Bedrock Batch and Flex, OpenAI Batch, Agent Platform Flex and Batch

Priority or fast

A premium per token

Seller

Bedrock Priority (75% more); OpenAI Fast (double)

Dedicated or provisioned

Per hour, used or not

Buyer

Bedrock Provisioned Throughput (no commitment, or 1 or 6 months); Foundry PTUs (reserved for a month or a year); Agent Platform (1 week to 1 year, a year 26% cheaper than a month)

Own servers, rented on a term or bought

Per hour or upfront, used or not

Buyer

See rent, buy or use an API

An inference provider renting whole servers on a term is in the buyer's seat: it pays for every hour and must fill them.

What an API saves a business

Most businesses want answers, not GPUs:

Job

If you run the model yourself

Through an API

Idle hours

You pay for the servers every hour, busy or not

Per-token pricing: only the tokens you use

Hardware

Buy or rent servers; one NVIDIA DGX B200 draws up to about 14.3 kW, needing data-centre power and cooling

None

Serving software

Install, tune and upgrade it: the vLLM serving engine shipped 21 minor versions in 2026

The provider's job

Security

A critical vLLM flaw, rated 9.8 out of 10 for severity, in February 2026, and 40 security advisories from 6 October to 18 November

The provider's job

On-call

Google's SRE book puts a single-site 24/7 rota at eight engineers minimum

The provider's job

New models

Download, test and re-tune each one

Change one name in your code

Data protection

Your own controls and audits

Bedrock: makers cannot see prompts; India-only processing for some OpenAI and Claude models. Foundry: the same for models sold by Azure. Google: no training on your data without permission

APIs churn too. Anthropic gives at least 60 days' notice before retiring a model, OpenAI at least six months for generally available ones, and OpenAI is closing self-serve fine-tuning (further training on your own data): new customers already cannot use it, and its remaining existing customers cannot start new jobs from 6 January 2027. Keeping processing in one region costs about 10% more at OpenAI (for models released from 5 March 2026) and at Anthropic (for US-only processing).

Among open-weight users in production, ETR (September 2026, via Techstrong.ai) found 46% self-manage in the cloud, 42% run on premises, 29% use hyperscalers and 27% managed inference providers; many use several.

How one GPU serves many customers

Serving engines such as vLLM, SGLang, TensorRT-LLM and NVIDIA Dynamo keep each GPU busy with many customers at once:

Technique

Plain meaning

Why it cuts cost

Continuous batching

New requests join the running batch at every step, like a bus that stops everywhere

Many customers share one GPU

Paged KV cache

The KV cache is the model's notes on what it has read, stored in small pages

Older systems wasted 60% to 80% of this memory, vLLM's PagedAttention under 4%, so more requests fit

Prefix caching

Repeated text, such as a fixed system prompt, is reused, not read again

Why cached input is cheap

Splitting prefill and decode

Reading the prompt needs compute; writing the answer needs memory speed (NVIDIA)

Each runs on GPUs tuned for it

Quantisation (FP8, FP4)

The model's numbers stored in 8 or 4 bits, not 16

More tokens per GPU; NVIDIA reports NVFP4 is about 1.8 times smaller than FP8, with 1% or less accuracy loss on DeepSeek-R1-0528

The gains stack: one GPU can handle tens of thousands of tokens a second, reading and writing together.

The unit economics, worked through

Apply the formula from our cost per token guide to one 8-GPU NVIDIA B200 server selling OpenAI's open-weight gpt-oss-120b.

One 8-GPU B200 server selling gpt-oss-120b (illustration)

Figure

Throughput per GPU at 50 tokens a second per user (InferenceX)

44,489 tokens a second, input plus output

Tokens per server-hour

About 1.28 billion (1.14 billion in, 142 million out)

Server price per hour

$32 ($4.00 per GPU-hour, an illustration, not our price)

Cost per million tokens if busy all the time, half the time, a quarter of the time

$0.025, $0.05, $0.10

Revenue per fully busy hour: cheapest, median and one of the dearest OpenRouter listings

About $56, $185, $505

Share of the day it must be busy to cover $32: cheapest, median, dearest

About 58%, 17%, 6%

Speed per user

Tokens a second per GPU, in plus out

Busy share needed: cheapest listing

Median listing

50 tokens a second

44,489

About 58%

About 17%

100 tokens a second

29,548

About 87%

About 26%

150 tokens a second

20,866

Never (would need 123%)

About 37%

  • Throughput: SemiAnalysis's InferenceX run of 31 May 2026 (B200, 4-bit, vLLM), counting input plus output tokens. Our tokens guide's MLPerf figures count only output, so the two costs per million are not comparable.
  • Each request: 8,000 tokens in, 1,000 out, random inputs, no caching; revenue scales with load.
  • OpenRouter prices for gpt-oss-120b per million tokens, input then output: cheapest $0.03 and $0.15, median $0.10 and $0.50, one of the dearest $0.35 and $0.75.
  • $4.00 per GPU-hour is a round illustration, not our price or a quote.
  • Staff, network, storage, software and platform fees are left out.

At full load, $32 of server time handles tokens that sell for $56 to $505. At the cheapest price, though, the server must be busy over half the day. And speed costs throughput: 50 tokens a second is already several times reading speed, and faster service per user means fewer users per GPU, which is our reading of why fast or priority tiers cost more.

Do inference providers make money?

Yes, when their GPUs stay busy. Published checks say the margin on a busy GPU is large:

Source, date

What was measured

Result

Catch

DeepSeek, 1 March 2025

24 hours of its own V3 and R1 serving on H800 GPUs, costed at $2 per GPU-hour

$87,072 of cost against $562,027 if all tokens were billed at R1 prices: a "cost profit margin" of 545% (profit 5.45 times cost)

Real revenue is much lower, DeepSeek says: free web and app use, cheaper V3, night discounts

InferenceX (SemiAnalysis), November 2026

Makers' own API prices against self-hosted cost on an AI-agent workload, GPUs costed as if owned at hyperscaler volume

GLM-5.3: API $0.303, self-hosted on B300 $0.030 per million tokens (10 times less). Kimi K3.1 Pro: 10 times less on B300, 1.8 times on B200 (different serving software on each)

Mostly cached input

Why the same model costs five times more

The pattern repeats across four popular open models (OpenRouter listings on 19 November 2026; counts include every tier and, for GLM, DeepSeek and Kimi, the maker's own listing):

Model

Listings (sellers)

Output price per million tokens

Median

Precision shown

GLM-5.3

49 (37)

$1.32 to $6.60

$4.40

18 fp4, 4 nvfp4, 12 fp8, 15 not stated

DeepSeek V4.1 Flash (1017)

50 (41)

$0.15 to $2.40

$0.69

23 fp8, 8 fp4 or mxfp4, 19 not stated

Kimi K3.1 Pro

27 (20)

$6 to $22

$10

17 fp4 or mxfp4, 1 fp8, 1 bf16, 8 not stated

gpt-oss-120b

24 (21)

$0.15 to $0.95

$0.50

10 fp4, 1 int4, 2 fp8, 4 bf16 or fp16, 7 not stated

  • Routing: OpenRouter shows each provider's own price and charges a fee when you buy credits. Its default routing weighs price after cache hits, speed, time to first token, rate limits, reliability and quality, so the cheapest seller does not take all the traffic.
  • Precision: fp4, nvfp4, mxfp4 and int4 are 4-bit formats, fp8 is 8-bit, and bf16 and fp16 are 16-bit; fewer bits mean more tokens per GPU. The cheapest gpt-oss-120b listings are 4-bit, but some 16-bit ones cost almost as little.
  • Quality: the same weights can give different results: OpenRouter launched Exacto, endpoints chosen for accurate tool use, and Kimi's maker, Moonshot AI, published a vendor verifier after results differed across providers.

The risks of selling tokens

  • Price wars: OpenRouter's average price paid for open-weight models fell 70%, from $0.28 to $0.084 per million tokens, between January and October 2026. Since May 2026, monthly tokens for GLM's maker Z.ai grew about 20 times, but spend only about 6 times.
  • Model churn: four GLM-5.x versions reached OpenRouter between 11 February and 18 August 2026. Software updates keep raising output after launch: DeepSeek V4 Pro on B200 gained about 1.7 times in 43 days (SemiAnalysis), and about 5 times on GB300 from April to June 2026 (SGLang). Providers that re-tune fast win; each new version restarts the work.
  • Idle capacity: DeepSeek serves on all its servers at the daytime peak and moves some to research and training at night; its inference fleet averaged about 82% of peak.
  • Licences: some open-weight licences need a separate agreement to sell the model as a service, such as Kimi K3.1 Pro once the seller passes $20 million in revenue in any 12 months, and Mistral Large 4 at any revenue; internal use is exempt (not legal advice).

When owning or renting GPUs wins

The buyer's break-even is the seller's, seen from the other side. OpenRouter's formula: GPU rent per hour ÷ API cost of one fully busy GPU-hour of tokens = the share of the day the GPUs must be busy. Our gpt-oss-120b server needs about 17% at the median API price and 58% at the cheapest, before staff. OpenRouter's own examples, which leave out engineering time, are tougher: DeepSeek V4.1 Flash on four H200s never broke even against the cheapest endpoint that keeps no data; on two B200s it needed 37% to 76% at the lower rent it tested, and at the higher rent it never matched the cheapest.

Self-hosting tends to win for:

  • steady, high volume of a large open model;
  • data that must stay on hardware you control (zero data retention, a provider's promise not to store prompts, is not that: prompts still run on its GPUs, OpenRouter notes);
  • models fine-tuned on your own data;
  • freezing a model version;
  • response times or a service guarantee you set yourself.

In India, the Reserve Bank of India requires payment-system data to be stored only in India, and SEBI, the markets regulator, requires regulated firms' cloud data to stay there; the DPDP Act, India's data protection law, has no blanket localisation rule. Not legal advice; self-hosting alone makes no one compliant.

API prices by model are in our sizing guide.

Selling tokens? Start with the server cost

Our side of the formula is the server: whole 8-GPU NVIDIA H100, H200, B200 or B300 machines, or GB300 NVL72 racks on request, from verified operators in North America, the UK, Europe, India, Asia Pacific and the Gulf, for at least one year, billed upfront. Quotes are per GPU-hour for comparison; we do not rent by the hour or sell tokens. Tell us the model, traffic and per-user speed you sell or need, and any data rules you must meet, and we will size the server and get quotes: ask for a quote, or see the servers we source. If an API already covers your volume and data rules, stay on it.

Where these figures come from

OpenRouter figures were pulled on 19 November 2026 and change daily. Tokens per hour, revenue and break-even shares are our arithmetic.

Frequently asked questions

What is an LLM inference provider?
A company that runs AI models on its own or rented GPU servers and sells the output through an API, priced per million tokens. Most serve open-weight models such as DeepSeek, GLM, Kimi or gpt-oss. Model makers and platforms such as Amazon Bedrock also sell this inference as a service.
How do inference providers sell tokens and make money?
They sell per million tokens through their own APIs and through aggregators such as OpenRouter, often with cheaper batch tiers, dearer priority tiers and dedicated capacity billed by the hour. Their GPU servers cost the same per hour, busy or idle, so profit depends on keeping them busy. In our illustration, an 8-GPU B200 server at $4.00 per GPU-hour serving gpt-oss-120b covers its hardware if busy about 17% of the day at the median OpenRouter price, or 58% at the cheapest.
Why does the same AI model cost five times more from one provider than another?
Sellers differ in hardware, number precision, speed, reliability and quality, and some charge more for priority tiers. On 19 November 2026, GLM-5.3 had 49 OpenRouter listings from 37 sellers, with output prices from $1.32 to $6.60 per million tokens.
What do fp4, fp8 and bf16 mean on OpenRouter?
They show the precision a provider serves: bf16 stores each of the model’s numbers in 16 bits, fp8 in 8 and fp4 in 4; nvfp4, mxfp4 and int4 are other 4-bit formats. Fewer bits mean less memory and more tokens per GPU, usually with a small loss of accuracy. “Unknown” means the provider has not said.
Do I need my own GPU to use AI in my business?
Usually not. An API gives you current models and pay-per-use pricing, with no hardware, patching or on-call staff. Owning or renting GPUs makes sense for steady, heavy traffic, data that must stay on hardware you control, models fine-tuned on your own data or a frozen model version.
When is self-hosting an LLM cheaper than an API?
Divide the hourly cost of your GPUs by what the same tokens would cost through an API if the GPUs were fully busy: that is the share of the day they must be busy to break even. In our November 2026 examples, before staff costs, it ranged from about 6% of the day (gpt-oss-120b on an 8-GPU B200 server against one of the dearest API listings; about 17% at the median price) to never.

Need GPU capacity?

Tell us your GPU count, term and region. We source the best-fit nodes across our provider network in North America, the UK, Europe, India, Asia Pacific and the Gulf, and confirm availability and lead time for your region before you order.