Will your model fit? A practical guide to GPU memory for AI

Before you rent or buy anything, one question decides everything else: will your model actually fit in the memory you have? It sounds like it should need a benchmark. It does not — it is arithmetic, and you can do it on the back of an envelope. This guide gives you the numbers, so you can size the hardware to the job instead of guessing.

The one number that governs everything

A model's memory footprint starts with a simple product: bytes per parameter × number of parameters. In half precision (FP16 or BF16, the normal way models run today) each parameter takes 2 bytes. So a rough weight footprint is just "two times the parameter count in billions, in gigabytes". A 7-billion-parameter model is about 14 GB of weights; 70 billion is about 140 GB; 405 billion is about 810 GB. Everything else — the KV cache, the optimizer, the activations — is added on top of that base.

Inference: what it takes to run a model

To serve a model you need room for the weights plus the KV cache — the running memory of the conversation, which grows with context length and how many requests you batch together. At short contexts the weights dominate; at long contexts the KV cache can rival them.

As a rule of thumb for FP16 inference:

• 7B — about 14 GB of weights. Fits on almost anything.
• 70B — about 140 GB of weights. Fits on a single B300 (288 GB) with plenty of headroom for a long-context KV cache.
• 405B — about 810 GB of weights. Needs several GPUs; comfortably within an 8-GPU node.

This is where memory per GPU earns its keep. The B300's 288 GB of HBM3e means a 70B-class model runs on one GPU rather than being split across several — simpler to serve, and lower latency.

Training: why it needs so much more

Training a model is far hungrier than running it, because you have to hold several copies of the numbers at once. For a full fine-tune with the usual Adam optimizer in mixed precision, budget roughly 16 to 20 bytes per parameter: the weights, the gradients, the optimizer's two running averages, a master copy of the weights, and then activations on top. That turns the inference figures into something much larger:

• 7B — roughly 120–160 GB. A single B300 handles it.
• 70B — roughly 1.1–1.4 TB. This is squarely 8-GPU-node territory; a full node is about 2.3 TB.

This is the honest reason full training of large models means a node, not a card: you are not memory-bound by the model alone, you are memory-bound by everything training drags along with it.

The levers that change the answer

The numbers above are the worst case. Several well-worn techniques cut them down, and knowing them can move your job from "needs a node" to "fits on one GPU":

• Quantization. Run the weights at lower precision. INT8 halves the memory (1 byte per parameter), INT4 quarters it (0.5 byte). A 70B model quantized to INT4 is about 35 GB to serve — small enough that even the largest open models fit on one B300 for inference.
• LoRA / QLoRA fine-tuning. Instead of training all the weights, freeze them and train small adapter layers. QLoRA goes further and quantizes the frozen base to 4-bit. Together they let you fine-tune a 70B model in roughly the memory it takes to run it — often on a single GPU.
• Gradient checkpointing. Trade compute for memory by recomputing activations during the backward pass instead of storing them. Slower, but it shrinks the activation budget that balloons at long sequence lengths.
• Context length and batch size. For inference these are the dials on the KV cache. Halve the context or the batch and you halve that part of the footprint.

A back-of-envelope sizing guide

Keep these four rules and you can size almost any job in your head (billions = the parameter count in billions):

• Serve, FP16: GB ≈ 2 × billions, plus about 20% for the KV cache.
• Serve, INT4: GB ≈ 0.5 × billions, plus a little overhead.
• Full fine-tune (Adam, mixed precision): GB ≈ 16–20 × billions.
• LoRA / QLoRA fine-tune: close to the inference figure, not the training one.

Mapping it to real hardware

Put the two halves together and the picture is clear. A single B300 gives you 288 GB; a full 8-GPU B300 node gives you about 2.3 TB. So:

• One B300 serves 70B-class models in FP16 with room to spare, or the very largest open models when quantized — and full-fine-tunes models up to roughly 10–15B.
• An 8-GPU node is what you want for full training of 70B-class models, or for serving the biggest open models at scale with long context.

If you already know your model and method, this tells you how many GPUs you need before you talk to anyone. If you are not sure, that is fine — tell us the model and what you want to do with it, and we will size it with you.

Getting started

Once you know the memory you need, the rest is straightforward. See what a job costs in our B300 cost breakdown, weigh renting against the alternatives in our rent, buy or API guide, or go straight to the B300 page and talk to an engineer. We would rather help you right-size the hardware than sell you more of it than your model needs.