NVIDIA GB200 NVL72 · one rack, 72 GPUs

Inside an AI Server

One question's trip through a Blackwell rack, with every number traced to a source.

GPUs72

HBM3e13.4 TB

NVLink130 TB/s

Power~120 kW

Drag to turn

Type a question. We'll follow it through the machine.

Loading the real tokenizer…

Part 1 · The journey01 / 03

Schematic · GB200 NVL72 reference hardware · drag to turn

Part 2 · The numbers02 / 03

Part 2 · The numbers

Change the machine, watch the physics

The story above used one setup. Here you choose the model, the chip, the number format and how many people share a copy. Every figure below is recomputed by the same simulator.

Your server

Model Hardware Number format GPUs per copy Prompt (tokens)

Answer (tokens)

People at once, per copy

01 / The numbers · The memory wall01 / 05

01 / The numbers · The memory wall

Speed is set by memory, not math

People sharing each step

02 / The numbers · The memory map02 / 05

Model weightsKV cacheFreeKept back by the runtime (10%)

KV cache per token, by model (FP8)per token · 128K-token chat

02 / The numbers · The memory map

Where all that memory goes

03 / The numbers · The price of a token03 / 05

03 / The numbers · The price of a token

Fast or cheap: pick a point on the curve

Your price per GPU-hour, as a share of each chip's on-demand list price

04 / The numbers · Power and heat04 / 05

04 / The numbers · Power and heat

Every token is a little heat

05 / The numbers · Chip generations05 / 05

Four years of NVIDIA data-center GPUs

Swipe the table sideways to see every column.

Part 3 · Your turn03 / 03

Your turn · Launch day50,000 people

Model Hardware Number format GPUs per copy

People per copy

Monthly bill

–

Your turn · Launch day

Serve 50,000 people at once for the lowest bill

Your chat app just went viral. 50,000 people are chatting at the same moment. Each message carries a 2,000-token prompt (instructions plus chat history) and gets a 500-token answer.

People leave if the first word takes longer than 2 seconds, or if text crawls in slower than 40 tokens per second. Pick a frontier-size model, a chip, a number format and how many people share each copy. The simulator tells you how many copies you need and what they cost at list price.

Hints: Hopper chips can't run FP4. Big mixture-of-experts models need a lot of memory before the first user arrives. More people per copy is cheaper until everyone gets too slow.

Method and notesHow the numbers were made