Skip to content

How many GPUs to run an LLM: the node each open-weight model needs

19 November 2026Vinod Chacko
How many GPUs to run an LLM: the node each open-weight model needs

Part of our AI models guide: 11 posts on choosing and running AI models.

Claude, GPT and Gemini cannot run on your own GPUs at any count: their makers keep the weights, so you can only rent them by the token. Open-weight models, whose weights (the model files) you can download, can, and how many GPUs one needs depends on the size of those files. A model under about 60 GB, such as gpt-oss-120b, fits on one 80 GB GPU, and up to about 110 GB on one 141 GB H200; up to about 450 GB takes 2 to 4 H200s (or a full 8x H100 node). From 450 GB to 900 GB, a model loads on one 8-GPU H200 server (a node), though published recipes run GLM-5.3 and DeepSeek V4-Pro on a B200 node for their full 1M-token context. The largest, about 1.5 TB at 4-bit, such as Kimi K3.1 Pro, need a full 8-GPU B300 node or two H200 or B200 nodes. Sizes checked on 19 November 2026.

Key terms in plain English

An LLM (large language model) is the kind of AI model behind chat assistants. Its weights, also called parameters, are the billions (B) or trillions (T) of numbers that make up the trained model. Whoever holds them can run it.

  • Closed-weight: the maker keeps the weights, so you use the model only through its API (a web service your software calls), paying per token, the smallest unit of text an AI model reads and writes, often a short word or part of one (see cost per token, explained).
  • Open-weight: the weights are published, so you can download the model and run it on your own or rented GPUs, under its licence.
  • Open source: by the Open Source Initiative's definition, the weights plus the full training and inference code and details of the training data; DeepSeek, Kimi and Qwen are open-weight only (see open-weight vs closed vs open source).
  • 8-GPU node: one server with eight data-centre GPUs joined by NVLink, NVIDIA's fast link inside the box (see PCIe, SXM, HGX and DGX).
  • Precision: how many bits store each number: 16 for BF16, 8 for FP8 and 4 for 4-bit formats such as NVFP4 and MXFP4. FP8 needs about half the memory of BF16, and 4-bit about a quarter.
  • Mixture-of-experts (MoE): a model that uses only a small active share of its parameters for each token, though all of them must sit in GPU memory. A dense model uses every parameter every time.
  • Context: how much text the model can consider at once. The KV cache is its working memory per conversation, and it grows with context length and the number of users.

How many GPUs each open model needs

Sizes are the weights only, from each model's card (its description page on Hugging Face) and file list. Hardware comes from serving recipes, tested setups published by NVIDIA and by vLLM, popular open-source software for running models. "Smallest setup" is a floor, not a recommendation: long context and many simultaneous users need extra memory for the KV cache. We arrange whole 8-GPU servers; a model that needs fewer GPUs can run as several copies on one server to serve more users.

Model (maker)

Parameters, total / active

Weights size (format)

Licence

Smallest setup that loads the weights

Source

gpt-oss-120b (OpenAI, US)

117B / 5.1B

About 60 to 65 GB (4-bit MXFP4)

Apache 2.0

One 80 GB GPU

Model card

Qwen3.8-27B (Alibaba, China)

27.8B, dense

About 56 GB (BF16)

Apache 2.0

One 80 GB GPU

Our arithmetic

Gemma 4 31B (Google, US)

31.3B, dense

About 63 GB (BF16)

Apache 2.0

One 80 GB GPU (tight with context; one H200 is comfortable)

Our arithmetic

DeepSeek V4.1 Flash, 1017 release (DeepSeek, China)

552B backbone (about 750B with memory tables) / 8B to 16B

About 511 GB (FP8 plus 4-bit)

MIT

One 8x H200 node, with room for context

vLLM recipe

Muse Spark Open (Meta, US)

About 816B / 42B

About 533 GB (NVFP4) or 1.64 TB (BF16)

Custom, gated

4-bit: one 8x B200 node. BF16: one 8x B300 node

Our arithmetic

GLM-5.3 (Z.ai, China)

About 753B / 39B (vLLM recipe)

About 756 GB (FP8)

Custom, MIT-based

One 8x H200 node; one 8x B200 node for the full 1M-token context

vLLM recipe

DeepSeek V4-Pro-0813 (DeepSeek, China)

1.6T to 1.7T / 49B

About 893 GB (FP8 plus 4-bit)

MIT

One 8x H200 node with context heavily capped; NVIDIA runs its own 4-bit build at the full 1M-token context on one 8x B200 node

NVIDIA recipe

Nemotron 3 Ultra (NVIDIA, US)

550B / 55B

About 1.1 TB (BF16); a 4-bit version is also published

OpenMDW 1.1

One 8x B200 node

Model card

Qwen3.8 2.4T (Alibaba, China)

2.4T / 95B

About 1.45 TB (NVFP4, NVIDIA's build per vLLM, not Alibaba's); about 2.5 TB (FP8)

Custom

4-bit: one 8x B300 node or two 8x H200 nodes. FP8: 16 B300s or 32 H200s

vLLM recipe

Kimi K3.1 Pro (Moonshot AI, China)

2.8T / 104B

About 1.56 TB (4-bit experts, BF16 elsewhere)

Custom

One 8x B300 node; two nodes on H200 or B200

vLLM recipe; two-node line our arithmetic

Claude, GPT, Gemini, Muse Wonder, Grok 4.7, Qwen4-Max

Not published

Not published

Proprietary

Cannot be self-hosted

API only

Notes. "Our arithmetic" compares the published file size with GPU memory. NVIDIA's DeepSeek recipe uses the April preview release (about 865 GB) and caps it at 86,016 tokens of context on 8x H200; for the full 1M-token context on 8x B200 it uses NVIDIA's own 4-bit build. Nemotron 3 Ultra's card also lists 8x H200 and 16x H100 as minimum setups; only its TensorRT-LLM support is Blackwell-only for now. Our B200 row follows the 80% rule. NVFP4 is a 4-bit format built for NVIDIA's newer Blackwell GPUs (B200 and B300), so Muse Spark Open's 4-bit row points to the B200.

Regional flagships are sized in their own guides, linked below. Europe's Mistral Large 4 (about 1.06 TB at FP8) fits one 8x B200 or B300 node. The UAE's Jais 3 and K2 Horizon, at 375B (about 760 GB in BF16), fit one 8x H200 node. India's Sarvam 105B (about 212 GB once loaded in BF16, our arithmetic; the published files are 32-bit, about 424 GB) fits four 80 GB GPUs.

The rule of thumb by size

Our rule of thumb: keep the weights to about 80% of GPU memory, leaving the rest for the KV cache; long contexts and many users need more. Our GPU memory guide shows the maths.

Weights size

Smallest single setup

Examples (19 November 2026)

Under about 110 GB

One GPU: about 60 GB on an 80 GB GPU, about 110 GB on an H200

gpt-oss-120b, Qwen3.8-27B, Gemma 4 31B

About 110 GB to 450 GB

2 to 4 H200s, part of one 8-GPU server (or a full 8x H100 node)

Sarvam 105B in BF16

About 450 GB to 900 GB

One 8x H200 node (1,128 GB)

DeepSeek V4.1 Flash, GLM-5.3, DeepSeek V4-Pro-0813 (short context)

About 900 GB to 1.15 TB

One 8x B200 node (1,440 GB)

Nemotron 3 Ultra (BF16), Mistral Large 4 (FP8); GLM-5.3 and DeepSeek V4-Pro (NVIDIA's 4-bit build) at full context

About 1.15 TB to 1.8 TB

One 8x B300 node (2,304 GB)

Kimi K3.1 Pro, Qwen3.8 2.4T (4-bit), Muse Spark Open (BF16)

Over about 1.8 TB

Two or more nodes, or a GB300 NVL72 rack

Qwen3.8 2.4T at FP8 or BF16

Why node memory decides it

GPU

Memory per GPU

Memory per 8-GPU node

Native 4-bit (FP4) maths

H100

80 GB

640 GB

No

H200

141 GB

1,128 GB

No

B200

180 GB

1,440 GB

Yes

B300

288 GB

2,304 GB

Yes

Blackwell GPUs (B200 and B300) do 4-bit maths natively; Hopper GPUs (H100 and H200) do not, so many 4-bit versions of models are released with Blackwell in mind (see Hopper vs Blackwell). Even so, memory sets the count: by vLLM's recipe, Qwen3.8 2.4T at 4-bit needs 8 B300s or 16 H200s, because both add up to about 2.3 TB. vLLM's Qwen recipe lists a B300 as 268 because it counts in GiB; that is the same 288 GB.

Past one node, the servers need a fast network between them: see NVLink and InfiniBand and renting a multi-node B300 cluster. For the largest models at scale, a GB300 NVL72 rack joins 72 GPUs in one NVLink domain (our NVL72 explainer). To choose between the two Blackwell nodes, see B300 vs B200.

Can I run Claude or GPT myself?

No. Claude (including Fable 5.2 and Opus 5.5), GPT-6.2, Gemini 4, Meta's Muse Wonder, Grok 4.7 and Alibaba's Qwen4-Max have no published weights, so any GPU count you see for them is a guess. To run a model on hardware you control, pick an open-weight one and test it on your own tasks.

Who makes what in November 2026

The US labs keep their best models closed, but most also publish smaller open-weight ones; the strongest open-weight models come from Chinese labs. Alibaba's new Qwen4-Max is API-only; Alibaba says smaller Qwen4 models will get open weights, with no date.

Maker (country)

Closed models (API only)

Open-weight models you can download

Licence of the open models

Anthropic (US)

Claude Fable 5.2, Opus 5.5, Sonnet 5.5, Haiku 5.5

None

Not applicable

OpenAI (US)

GPT-6.2 Astra, GPT-6.2 Sol, GPT-6 Luna

gpt-oss-120b, gpt-oss-20b (August 2025)

Apache 2.0

Google (US)

Gemini 4 Carbon, Neon, Argon, Hydrogen

Gemma 4 (up to 31B)

Apache 2.0

Meta (US)

Muse Wonder, Muse Spark

Muse Spark Open, Llama 4

Muse Model License (custom, gated); Llama 4 Community License

xAI (US)

Grok 4.7

Older Grok-1 and Grok-2 only

Read each release's licence

NVIDIA (US)

None listed

Nemotron 3 Ultra

OpenMDW 1.1

DeepSeek (China)

None

DeepSeek V4-Pro-0813, DeepSeek V4.1 Flash

MIT

Moonshot AI (China)

None

Kimi K3.1 Pro

Custom Kimi licence

Z.ai (China)

None

GLM-5.3, GLM-5.3-Flash

Custom GLM-5.3 licence; read Flash's own file

Alibaba (China)

Qwen4-Max

Qwen3.8 2.4T, Qwen3.8-27B

Custom Qwen3.8-Max licence (2.4T); Apache 2.0 (27B)

How far behind are open models?

On Artificial Analysis's Intelligence Index (version 5), which combines many independent tests, Claude Fable 5.2 (at its maximum reasoning setting) leads at 65. The best open-weight model, Kimi K3.1 Pro, scores 46 and ranks 12th: a 19-point gap. About 60% of version 5 is on tests the models have not seen before, and Artificial Analysis says scores from different versions are not meant to be compared. Arena's text leaderboard, which ranks models by people's votes, agrees: when we checked, all of its top 15 were closed-weight.

So open models win on price, not peak quality, which is why they carry most of OpenRouter's traffic.

Beyond the US and China

Our guides to US AI models and Chinese AI models cover the two leaders in depth. Other regions build their own models, mainly for language coverage, data rules and keeping the model in the country. India's flagship is Sarvam 105B (Apache 2.0), which Sarvam says it trained on IndiaAI Mission compute (see Indian AI models). In the UAE, Jais 3 covers Arabic and English and is built on MBZUAI's K2 Horizon, which its makers describe as fully open: weights, training data and code. TII's Falcon family also comes from Abu Dhabi; its multilingual Falcon-H1 models are open-weight, but its dedicated Arabic models have no public weights (see Arabic LLMs). In Europe, France's Mistral AI publishes open-weight models, and the EU AI Act adds rules of its own (see Mistral and European models). Japan and Korea have their own model makers too (see Japanese and Korean models).

What OpenRouter's data shows

OpenRouter is one API that reaches more than 600 models from over 80 providers, by its own count. For each request, it picks one of the providers serving the model and switches if one fails. It passes through each provider's listed token price and charges a 5.5% fee when you buy credits ($0.80 minimum). Stripe agreed to buy OpenRouter in August 2026.

OpenRouter's own data shows where developer traffic has gone:

OpenRouter, 2026

Open-weight models

Closed-weight models

Share of paid tokens, mid-October

Over 70%

Under 30%

Share of paid spend, October

About 25%

About 75%

Share of paid spend, January

11%

89%

In OpenRouter's latest weekly rankings, 8 of its 10 most-used models came from Chinese labs, led by DeepSeek V4.1 Flash (1017 release) with 62.1 trillion tokens. Agents, software that calls models in long loops, sent 76% of October's tokens. Agent prompts tend to be long, so plan memory for long context.

Read these numbers with care: OpenRouter's users lean towards developers and price-sensitive apps, many enterprises buy direct, and tokens are not spend. Menlo Ventures' November 2026 enterprise AI report put open-weight models (Menlo calls them open-source) at 19% of token usage, a share that draws partly on OpenRouter data, but 4% of API spend (Menlo invests in Anthropic and OpenRouter).

Open-weight is not free to resell

If you plan to sell tokens, the licence matters as much as the GPUs. Model-as-a-service (MaaS) means selling access to a model, usually through an API. Licences can differ by size within one family: Qwen3.8-27B is Apache 2.0, while Qwen3.8 2.4T has its own terms. Revenue tests count your company and its affiliates together, and under some licences, such as Qwen's, the extra terms then cover any commercial use of the model.

Model

Licence

Revenue trigger

Display rule

Internal-only use

DeepSeek V4-Pro, V4.1 Flash

MIT (permissive)

None

Keep the licence notice

Allowed

gpt-oss, Gemma 4, Qwen3.8-27B

Apache 2.0 (permissive)

None

Keep the licence notice

Allowed

Nemotron 3 Ultra

OpenMDW 1.1 (commercial use allowed)

None

Read the file

Allowed

GLM-5.3

Custom, MIT-based

MaaS company over $10B revenue in any 12 months: Z.ai security review

Read the file

Allowed

Kimi K3.1 Pro

Custom Kimi licence

MaaS company over $20M revenue in any 12 months: separate agreement with Moonshot

Show the model name above 100M monthly users or $20M monthly revenue

Exempt from the MaaS and display clauses

Qwen3.8 2.4T

Qwen3.8-Max Licence

MaaS or AI work-assistant company over $50M revenue in any 12 months: separate licence

Show the model name above 100M monthly users or $20M monthly revenue

Exempt from the separate-licence clause

Muse Spark Open

Muse Model License (custom, gated)

MaaS company over $20M revenue in any 12 months: separate agreement with Meta

"Built with Muse Spark Open"

Exempt, as are features built into your own product

Qwen's and GLM's licences say that simply relaying requests to models hosted by others is not MaaS. This is general information, not legal advice: read the LICENSE file for the exact model and size you deploy.

When running your own model pays

Teams run open-weight models on their own GPUs for four reasons:

  • Data stays on hardware they control, in the country they choose.
  • Steady, high volume.
  • Fine-tuning on their own data (see B300 fine-tuning cost).
  • Version control: the model does not change under them.

API prices per million tokens, for comparison, as of 19 November 2026:

Model

Weights

Input

Output

Basis

Claude Fable 5.2

Closed

$8

$40

Anthropic list price

GPT-6.2 Astra

Closed

$8

$40

OpenAI list price

Claude Opus 5.5

Closed

$4

$20

Anthropic list price

Gemini 4 Carbon

Closed

$2 ($4 from 1 December 2026)

$10 ($20 from 1 December 2026)

Google list price

Kimi K3.1 Pro

Open

$2.00

$10.00

Median of 27 provider listings on OpenRouter

GLM-5.3

Open

$1.26

$4.40

Median of 49 OpenRouter listings from 37 providers

DeepSeek V4-Pro-0813

Open

$1.32

$3.96

Median of 26 provider listings

DeepSeek V4.1 Flash (1017)

Open

$0.16

$0.69

Median of 50 provider listings

gpt-oss-120b

Open

$0.10

$0.50

Median of 24 provider listings

A rented node is a fixed cost per hour, busy or idle, so self-hosting pays when a large open model has steady, heavy traffic that keeps the node busy. For small, cheap models such as gpt-oss-120b, the API usually costs less.

In ETR's September 2026 survey of 200 enterprises, as reported by Techstrong.ai, 69% of those running open-weight models in production cited cost savings, and half of those still piloting named self-hosting cost as an obstacle. Our guides to renting, buying or using an API and GPU cost per hour go through the numbers.

Data control and compliance

Running open weights yourself means prompts and outputs never reach the model maker's API; they stay on hardware you choose (see bare metal vs cloud GPUs). It does not change how a model behaves, so test it before production. Some buyers rule out certain models for policy reasons; US federal agencies, defence contractors and regulated firms should take legal advice. The regional guides linked above each cover their own data rules.

Size your node in four steps

  1. Pick the model and the precision you will serve (4-bit, FP8 or BF16), and test it through an API at that precision (OpenRouter shows the precision each provider serves).
  2. Find the weights size in our table or in the model's file list on Hugging Face.
  3. Add memory for context length times simultaneous users; our GPU memory guide shows how.
  4. Choose the smallest node where the weights stay under about 80% of memory. If none fits, plan for more than one node and check the network between them (see multi-node B300 clusters and questions to ask a GPU provider).

Have us check your sizing

We arrange only whole 8-GPU servers (H100, H200, B200 or B300) from verified operators in six regions: North America, the UK, Europe, India, Asia Pacific and the Gulf, for at least a year, billed upfront, not by the hour. For comparison, an 8x B300 node is from $36 an hour ($4.50 per GPU-hour) on our GPU price list, the floor on our longest term, USD before tax; a one-year term costs more.

Give us the model, context length and traffic, and we will size and quote the node.

Where these figures come from

Checked on 19 November 2026 unless a date is given.

Frequently asked questions

How many GPUs do you need to run an LLM?
It depends on the size of the model’s weights: as of November 2026, open-weight models under about 60 GB, such as gpt-oss-120b, run on one 80 GB GPU, and up to about 110 GB on one 141 GB H200. Most models of 450 GB to 900 GB fit one 8-GPU H200 node, and the largest, about 1.5 TB, such as Kimi K3.1 Pro, need an 8-GPU B300 node or two H200 or B200 nodes.
How many GPUs do you need to run Kimi K3 (K3.1 Pro)?
Kimi K3.1 Pro’s 4-bit weights take about 1.56 TB, so the smallest single server that loads them is one 8-GPU B300 node, with 2,304 GB of GPU memory. They do not fit one 8x H200 node (1,128 GB) or one 8x B200 node (1,440 GB), so on those GPUs you need two nodes. Long context and many simultaneous users need more memory on top.
How many GPUs do you need to run DeepSeek V4?
DeepSeek V4-Pro’s official 0813 weights are about 893 GB. NVIDIA’s published serving recipe runs the April preview release on one 8x H200 node with context capped at about 86,000 tokens, and NVIDIA’s own 4-bit build on one 8x B200 node with the full 1M-token context. The smaller DeepSeek V4.1 Flash (1017 release) is about 511 GB and fits one 8x H200 node with room for context.
How many GPUs do you need to run GLM-5.3?
GLM-5.3 ships in FP8 at about 756 GB, and vLLM’s published serving recipe runs it on one 8x H200 node. For its full 1M-token context, the recipe uses one 8x B200 node.
How many GPUs do you need to run Qwen3.8?
A 4-bit NVFP4 build of Qwen3.8 2.4T (NVIDIA’s, per vLLM, not Alibaba’s own) is about 1.45 TB, so the minimum in vLLM’s published serving recipe is one 8x B300 node or two 8x H200 nodes; Alibaba’s FP8 weights, about 2.5 TB, need 16 B300s or 32 H200s. The smaller Qwen3.8-27B (about 56 GB, Apache 2.0) runs on one 80 GB GPU.
Can I run Claude, GPT or Gemini on my own GPUs?
No. Claude, GPT and Gemini are closed-weight models: their makers do not release the weights, so no number of GPUs lets you run them yourself, and you can only use them through an API.
Is open-weight the same as open source?
No. Under the Open Source Initiative’s definition, an open-source AI model must publish its weights plus the full training and inference code and detailed information about its training data. Most popular open models, including DeepSeek, Kimi and Qwen, publish only the weights, so they are open-weight.
Can I resell tokens from an open-weight model?
It depends on the licence: MIT and Apache 2.0 models such as DeepSeek V4, gpt-oss and Gemma 4 allow it. Kimi K3.1 Pro needs a separate agreement once a company running a model-as-a-service business passes $20M combined revenue in any 12 months, and Qwen3.8 2.4T once it passes $50M. Read the licence file for the exact model you deploy; this is not legal advice.
When is running your own model cheaper than paying per token?
Usually when you serve a large open model with steady, heavy traffic, because a rented node costs the same per hour whether it is busy or idle. For small, cheap models, API prices are often lower than the cost of keeping your own GPUs busy. Switching from a closed model only saves money if an open model meets your quality bar, so test it on your own tasks first.

Need GPU capacity?

Tell us your GPU count, term and region. We source the best-fit nodes across our provider network in North America, the UK, Europe, India, Asia Pacific and the Gulf, and confirm availability and lead time for your region before you order.