How many GPUs to run an LLM: the node each open-weight model needs

Part of our AI models guide: 11 posts on choosing and running AI models.
Claude, GPT and Gemini cannot run on your own GPUs at any count: their makers keep the weights, so you can only rent them by the token. Open-weight models, whose weights (the model files) you can download, can, and how many GPUs one needs depends on the size of those files. A model under about 60 GB, such as gpt-oss-120b, fits on one 80 GB GPU, and up to about 110 GB on one 141 GB H200; up to about 450 GB takes 2 to 4 H200s (or a full 8x H100 node). From 450 GB to 900 GB, a model loads on one 8-GPU H200 server (a node), though published recipes run GLM-5.3 and DeepSeek V4-Pro on a B200 node for their full 1M-token context. The largest, about 1.5 TB at 4-bit, such as Kimi K3.1 Pro, need a full 8-GPU B300 node or two H200 or B200 nodes. Sizes checked on 19 November 2026.
Key terms in plain English
An LLM (large language model) is the kind of AI model behind chat assistants. Its weights, also called parameters, are the billions (B) or trillions (T) of numbers that make up the trained model. Whoever holds them can run it.
- Closed-weight: the maker keeps the weights, so you use the model only through its API (a web service your software calls), paying per token, the smallest unit of text an AI model reads and writes, often a short word or part of one (see cost per token, explained).
- Open-weight: the weights are published, so you can download the model and run it on your own or rented GPUs, under its licence.
- Open source: by the Open Source Initiative's definition, the weights plus the full training and inference code and details of the training data; DeepSeek, Kimi and Qwen are open-weight only (see open-weight vs closed vs open source).
- 8-GPU node: one server with eight data-centre GPUs joined by NVLink, NVIDIA's fast link inside the box (see PCIe, SXM, HGX and DGX).
- Precision: how many bits store each number: 16 for BF16, 8 for FP8 and 4 for 4-bit formats such as NVFP4 and MXFP4. FP8 needs about half the memory of BF16, and 4-bit about a quarter.
- Mixture-of-experts (MoE): a model that uses only a small active share of its parameters for each token, though all of them must sit in GPU memory. A dense model uses every parameter every time.
- Context: how much text the model can consider at once. The KV cache is its working memory per conversation, and it grows with context length and the number of users.
How many GPUs each open model needs
Sizes are the weights only, from each model's card (its description page on Hugging Face) and file list. Hardware comes from serving recipes, tested setups published by NVIDIA and by vLLM, popular open-source software for running models. "Smallest setup" is a floor, not a recommendation: long context and many simultaneous users need extra memory for the KV cache. We arrange whole 8-GPU servers; a model that needs fewer GPUs can run as several copies on one server to serve more users.
Model (maker) | Parameters, total / active | Weights size (format) | Licence | Smallest setup that loads the weights | Source |
|---|---|---|---|---|---|
gpt-oss-120b (OpenAI, US) | 117B / 5.1B | About 60 to 65 GB (4-bit MXFP4) | Apache 2.0 | One 80 GB GPU | Model card |
Qwen3.8-27B (Alibaba, China) | 27.8B, dense | About 56 GB (BF16) | Apache 2.0 | One 80 GB GPU | Our arithmetic |
Gemma 4 31B (Google, US) | 31.3B, dense | About 63 GB (BF16) | Apache 2.0 | One 80 GB GPU (tight with context; one H200 is comfortable) | Our arithmetic |
DeepSeek V4.1 Flash, 1017 release (DeepSeek, China) | 552B backbone (about 750B with memory tables) / 8B to 16B | About 511 GB (FP8 plus 4-bit) | MIT | One 8x H200 node, with room for context | |
Muse Spark Open (Meta, US) | About 816B / 42B | About 533 GB (NVFP4) or 1.64 TB (BF16) | Custom, gated | 4-bit: one 8x B200 node. BF16: one 8x B300 node | Our arithmetic |
GLM-5.3 (Z.ai, China) | About 753B / 39B (vLLM recipe) | About 756 GB (FP8) | Custom, MIT-based | One 8x H200 node; one 8x B200 node for the full 1M-token context | |
DeepSeek V4-Pro-0813 (DeepSeek, China) | 1.6T to 1.7T / 49B | About 893 GB (FP8 plus 4-bit) | MIT | One 8x H200 node with context heavily capped; NVIDIA runs its own 4-bit build at the full 1M-token context on one 8x B200 node | |
Nemotron 3 Ultra (NVIDIA, US) | 550B / 55B | About 1.1 TB (BF16); a 4-bit version is also published | OpenMDW 1.1 | One 8x B200 node | Model card |
Qwen3.8 2.4T (Alibaba, China) | 2.4T / 95B | About 1.45 TB (NVFP4, NVIDIA's build per vLLM, not Alibaba's); about 2.5 TB (FP8) | Custom | 4-bit: one 8x B300 node or two 8x H200 nodes. FP8: 16 B300s or 32 H200s | |
Kimi K3.1 Pro (Moonshot AI, China) | 2.8T / 104B | About 1.56 TB (4-bit experts, BF16 elsewhere) | Custom | One 8x B300 node; two nodes on H200 or B200 | vLLM recipe; two-node line our arithmetic |
Claude, GPT, Gemini, Muse Wonder, Grok 4.7, Qwen4-Max | Not published | Not published | Proprietary | Cannot be self-hosted | API only |
Notes. "Our arithmetic" compares the published file size with GPU memory. NVIDIA's DeepSeek recipe uses the April preview release (about 865 GB) and caps it at 86,016 tokens of context on 8x H200; for the full 1M-token context on 8x B200 it uses NVIDIA's own 4-bit build. Nemotron 3 Ultra's card also lists 8x H200 and 16x H100 as minimum setups; only its TensorRT-LLM support is Blackwell-only for now. Our B200 row follows the 80% rule. NVFP4 is a 4-bit format built for NVIDIA's newer Blackwell GPUs (B200 and B300), so Muse Spark Open's 4-bit row points to the B200.
Regional flagships are sized in their own guides, linked below. Europe's Mistral Large 4 (about 1.06 TB at FP8) fits one 8x B200 or B300 node. The UAE's Jais 3 and K2 Horizon, at 375B (about 760 GB in BF16), fit one 8x H200 node. India's Sarvam 105B (about 212 GB once loaded in BF16, our arithmetic; the published files are 32-bit, about 424 GB) fits four 80 GB GPUs.
The rule of thumb by size
Our rule of thumb: keep the weights to about 80% of GPU memory, leaving the rest for the KV cache; long contexts and many users need more. Our GPU memory guide shows the maths.
Weights size | Smallest single setup | Examples (19 November 2026) |
|---|---|---|
Under about 110 GB | One GPU: about 60 GB on an 80 GB GPU, about 110 GB on an H200 | gpt-oss-120b, Qwen3.8-27B, Gemma 4 31B |
About 110 GB to 450 GB | 2 to 4 H200s, part of one 8-GPU server (or a full 8x H100 node) | Sarvam 105B in BF16 |
About 450 GB to 900 GB | One 8x H200 node (1,128 GB) | DeepSeek V4.1 Flash, GLM-5.3, DeepSeek V4-Pro-0813 (short context) |
About 900 GB to 1.15 TB | One 8x B200 node (1,440 GB) | Nemotron 3 Ultra (BF16), Mistral Large 4 (FP8); GLM-5.3 and DeepSeek V4-Pro (NVIDIA's 4-bit build) at full context |
About 1.15 TB to 1.8 TB | One 8x B300 node (2,304 GB) | Kimi K3.1 Pro, Qwen3.8 2.4T (4-bit), Muse Spark Open (BF16) |
Over about 1.8 TB | Two or more nodes, or a GB300 NVL72 rack | Qwen3.8 2.4T at FP8 or BF16 |
Why node memory decides it
GPU | Memory per GPU | Memory per 8-GPU node | Native 4-bit (FP4) maths |
|---|---|---|---|
80 GB | 640 GB | No | |
141 GB | 1,128 GB | No | |
180 GB | 1,440 GB | Yes | |
288 GB | 2,304 GB | Yes |
Blackwell GPUs (B200 and B300) do 4-bit maths natively; Hopper GPUs (H100 and H200) do not, so many 4-bit versions of models are released with Blackwell in mind (see Hopper vs Blackwell). Even so, memory sets the count: by vLLM's recipe, Qwen3.8 2.4T at 4-bit needs 8 B300s or 16 H200s, because both add up to about 2.3 TB. vLLM's Qwen recipe lists a B300 as 268 because it counts in GiB; that is the same 288 GB.
Past one node, the servers need a fast network between them: see NVLink and InfiniBand and renting a multi-node B300 cluster. For the largest models at scale, a GB300 NVL72 rack joins 72 GPUs in one NVLink domain (our NVL72 explainer). To choose between the two Blackwell nodes, see B300 vs B200.
Can I run Claude or GPT myself?
No. Claude (including Fable 5.2 and Opus 5.5), GPT-6.2, Gemini 4, Meta's Muse Wonder, Grok 4.7 and Alibaba's Qwen4-Max have no published weights, so any GPU count you see for them is a guess. To run a model on hardware you control, pick an open-weight one and test it on your own tasks.
Who makes what in November 2026
The US labs keep their best models closed, but most also publish smaller open-weight ones; the strongest open-weight models come from Chinese labs. Alibaba's new Qwen4-Max is API-only; Alibaba says smaller Qwen4 models will get open weights, with no date.
Maker (country) | Closed models (API only) | Open-weight models you can download | Licence of the open models |
|---|---|---|---|
Anthropic (US) | Claude Fable 5.2, Opus 5.5, Sonnet 5.5, Haiku 5.5 | None | Not applicable |
OpenAI (US) | GPT-6.2 Astra, GPT-6.2 Sol, GPT-6 Luna | gpt-oss-120b, gpt-oss-20b (August 2025) | Apache 2.0 |
Google (US) | Gemini 4 Carbon, Neon, Argon, Hydrogen | Gemma 4 (up to 31B) | Apache 2.0 |
Meta (US) | Muse Wonder, Muse Spark | Muse Spark Open, Llama 4 | Muse Model License (custom, gated); Llama 4 Community License |
xAI (US) | Grok 4.7 | Older Grok-1 and Grok-2 only | Read each release's licence |
NVIDIA (US) | None listed | Nemotron 3 Ultra | OpenMDW 1.1 |
DeepSeek (China) | None | DeepSeek V4-Pro-0813, DeepSeek V4.1 Flash | MIT |
Moonshot AI (China) | None | Kimi K3.1 Pro | Custom Kimi licence |
Z.ai (China) | None | GLM-5.3, GLM-5.3-Flash | Custom GLM-5.3 licence; read Flash's own file |
Alibaba (China) | Qwen4-Max | Qwen3.8 2.4T, Qwen3.8-27B | Custom Qwen3.8-Max licence (2.4T); Apache 2.0 (27B) |
How far behind are open models?
On Artificial Analysis's Intelligence Index (version 5), which combines many independent tests, Claude Fable 5.2 (at its maximum reasoning setting) leads at 65. The best open-weight model, Kimi K3.1 Pro, scores 46 and ranks 12th: a 19-point gap. About 60% of version 5 is on tests the models have not seen before, and Artificial Analysis says scores from different versions are not meant to be compared. Arena's text leaderboard, which ranks models by people's votes, agrees: when we checked, all of its top 15 were closed-weight.
So open models win on price, not peak quality, which is why they carry most of OpenRouter's traffic.
Beyond the US and China
Our guides to US AI models and Chinese AI models cover the two leaders in depth. Other regions build their own models, mainly for language coverage, data rules and keeping the model in the country. India's flagship is Sarvam 105B (Apache 2.0), which Sarvam says it trained on IndiaAI Mission compute (see Indian AI models). In the UAE, Jais 3 covers Arabic and English and is built on MBZUAI's K2 Horizon, which its makers describe as fully open: weights, training data and code. TII's Falcon family also comes from Abu Dhabi; its multilingual Falcon-H1 models are open-weight, but its dedicated Arabic models have no public weights (see Arabic LLMs). In Europe, France's Mistral AI publishes open-weight models, and the EU AI Act adds rules of its own (see Mistral and European models). Japan and Korea have their own model makers too (see Japanese and Korean models).
What OpenRouter's data shows
OpenRouter is one API that reaches more than 600 models from over 80 providers, by its own count. For each request, it picks one of the providers serving the model and switches if one fails. It passes through each provider's listed token price and charges a 5.5% fee when you buy credits ($0.80 minimum). Stripe agreed to buy OpenRouter in August 2026.
OpenRouter's own data shows where developer traffic has gone:
OpenRouter, 2026 | Open-weight models | Closed-weight models |
|---|---|---|
Share of paid tokens, mid-October | Over 70% | Under 30% |
Share of paid spend, October | About 25% | About 75% |
Share of paid spend, January | 11% | 89% |
In OpenRouter's latest weekly rankings, 8 of its 10 most-used models came from Chinese labs, led by DeepSeek V4.1 Flash (1017 release) with 62.1 trillion tokens. Agents, software that calls models in long loops, sent 76% of October's tokens. Agent prompts tend to be long, so plan memory for long context.
Read these numbers with care: OpenRouter's users lean towards developers and price-sensitive apps, many enterprises buy direct, and tokens are not spend. Menlo Ventures' November 2026 enterprise AI report put open-weight models (Menlo calls them open-source) at 19% of token usage, a share that draws partly on OpenRouter data, but 4% of API spend (Menlo invests in Anthropic and OpenRouter).
Open-weight is not free to resell
If you plan to sell tokens, the licence matters as much as the GPUs. Model-as-a-service (MaaS) means selling access to a model, usually through an API. Licences can differ by size within one family: Qwen3.8-27B is Apache 2.0, while Qwen3.8 2.4T has its own terms. Revenue tests count your company and its affiliates together, and under some licences, such as Qwen's, the extra terms then cover any commercial use of the model.
Model | Licence | Revenue trigger | Display rule | Internal-only use |
|---|---|---|---|---|
DeepSeek V4-Pro, V4.1 Flash | MIT (permissive) | None | Keep the licence notice | Allowed |
gpt-oss, Gemma 4, Qwen3.8-27B | Apache 2.0 (permissive) | None | Keep the licence notice | Allowed |
Nemotron 3 Ultra | OpenMDW 1.1 (commercial use allowed) | None | Read the file | Allowed |
GLM-5.3 | Custom, MIT-based | MaaS company over $10B revenue in any 12 months: Z.ai security review | Read the file | Allowed |
Kimi K3.1 Pro | Custom Kimi licence | MaaS company over $20M revenue in any 12 months: separate agreement with Moonshot | Show the model name above 100M monthly users or $20M monthly revenue | Exempt from the MaaS and display clauses |
Qwen3.8 2.4T | Qwen3.8-Max Licence | MaaS or AI work-assistant company over $50M revenue in any 12 months: separate licence | Show the model name above 100M monthly users or $20M monthly revenue | Exempt from the separate-licence clause |
Muse Spark Open | Muse Model License (custom, gated) | MaaS company over $20M revenue in any 12 months: separate agreement with Meta | "Built with Muse Spark Open" | Exempt, as are features built into your own product |
Qwen's and GLM's licences say that simply relaying requests to models hosted by others is not MaaS. This is general information, not legal advice: read the LICENSE file for the exact model and size you deploy.
When running your own model pays
Teams run open-weight models on their own GPUs for four reasons:
- Data stays on hardware they control, in the country they choose.
- Steady, high volume.
- Fine-tuning on their own data (see B300 fine-tuning cost).
- Version control: the model does not change under them.
API prices per million tokens, for comparison, as of 19 November 2026:
Model | Weights | Input | Output | Basis |
|---|---|---|---|---|
Claude Fable 5.2 | Closed | $8 | $40 | Anthropic list price |
GPT-6.2 Astra | Closed | $8 | $40 | OpenAI list price |
Claude Opus 5.5 | Closed | $4 | $20 | Anthropic list price |
Gemini 4 Carbon | Closed | $2 ($4 from 1 December 2026) | $10 ($20 from 1 December 2026) | Google list price |
Kimi K3.1 Pro | Open | $2.00 | $10.00 | Median of 27 provider listings on OpenRouter |
GLM-5.3 | Open | $1.26 | $4.40 | Median of 49 OpenRouter listings from 37 providers |
DeepSeek V4-Pro-0813 | Open | $1.32 | $3.96 | Median of 26 provider listings |
DeepSeek V4.1 Flash (1017) | Open | $0.16 | $0.69 | Median of 50 provider listings |
gpt-oss-120b | Open | $0.10 | $0.50 | Median of 24 provider listings |
A rented node is a fixed cost per hour, busy or idle, so self-hosting pays when a large open model has steady, heavy traffic that keeps the node busy. For small, cheap models such as gpt-oss-120b, the API usually costs less.
In ETR's September 2026 survey of 200 enterprises, as reported by Techstrong.ai, 69% of those running open-weight models in production cited cost savings, and half of those still piloting named self-hosting cost as an obstacle. Our guides to renting, buying or using an API and GPU cost per hour go through the numbers.
Data control and compliance
Running open weights yourself means prompts and outputs never reach the model maker's API; they stay on hardware you choose (see bare metal vs cloud GPUs). It does not change how a model behaves, so test it before production. Some buyers rule out certain models for policy reasons; US federal agencies, defence contractors and regulated firms should take legal advice. The regional guides linked above each cover their own data rules.
Size your node in four steps
- Pick the model and the precision you will serve (4-bit, FP8 or BF16), and test it through an API at that precision (OpenRouter shows the precision each provider serves).
- Find the weights size in our table or in the model's file list on Hugging Face.
- Add memory for context length times simultaneous users; our GPU memory guide shows how.
- Choose the smallest node where the weights stay under about 80% of memory. If none fits, plan for more than one node and check the network between them (see multi-node B300 clusters and questions to ask a GPU provider).
Have us check your sizing
We arrange only whole 8-GPU servers (H100, H200, B200 or B300) from verified operators in six regions: North America, the UK, Europe, India, Asia Pacific and the Gulf, for at least a year, billed upfront, not by the hour. For comparison, an 8x B300 node is from $36 an hour ($4.50 per GPU-hour) on our GPU price list, the floor on our longest term, USD before tax; a one-year term costs more.
Give us the model, context length and traffic, and we will size and quote the node.
Where these figures come from
Checked on 19 November 2026 unless a date is given.
- Model sizes, parameters and context: the Hugging Face model cards and file lists linked in the first table, plus the Muse Spark Open 4-bit files, xAI, Sarvam 105B, K2 Horizon 375B, Jais 3 375B, TII on Hugging Face (Falcon releases) and Mistral Large 4. DeepSeek V4-Pro-0813: 66 weight files, about 893 GB; 49B active per the April release card. Size bands and "our arithmetic" rows are FlexiCloud's calculations.
- Serving recipes: vLLM recipes for DeepSeek V4.1 Flash, GLM-5.3, Qwen3.8 2.4T and Kimi K3.1 Pro; NVIDIA Dynamo recipe for DeepSeek V4-Pro.
- Licences: the LICENSE files for Kimi K3.1 Pro, Qwen3.8 2.4T and GLM-5.3, plus each model card above. Open source definition: Open Source AI Definition 1.0 (October 2024).
- Releases: Anthropic, Claude Fable 5.2 (9 November 2026), OpenAI, GPT-6.2 Astra and Sol (17 November 2026), Google DeepMind blog (Gemini 4), The Next Web, Qwen4-Max (16 November 2026), MBZUAI, K2 Horizon (3 September 2026).
- Quality: Artificial Analysis leaderboard (Intelligence Index v5) and its v5 launch note (27 October 2026); Arena text leaderboard (18 November 2026).
- OpenRouter: About, FAQ (fees and routing), October 2026 data, rankings (week to 18 November 2026) and model listings (provider counts and medians, our calculation). Ownership: Stripe and OpenRouter (19 August 2026).
- List prices: Anthropic, OpenAI and Google.
- Surveys: Menlo Ventures enterprise AI report (November 2026) and Techstrong.ai on ETR's survey (September 2026).