VRAM Match

v7.3: a third measurement on the same RTX 5080, and the first number I have on what the cliff actually costs. The 5080 owner in the thread answered both of my open questions and gave me his hardest row: Qwen3.5-27B IQ4_XS, 14,689,289,728 bytes, 32K, 49.4 tok/s, 585 MiB free with the display moved to the motherboard output. That is 75.6% of his card's roofline, and his row now reproduces end to end: 13.68 GiB of weights plus 1.00 GiB of q8_0 cache at 32K (16 full-attention layers x 4 KV heads x head_dim 256) plus the 0.4 GiB runtime lands within 20 MiB of the 15.16 GiB he measured the model using. My efficiency band for Turing and newer was already 62-78% from his first two rows; a third row at 75.6% does not move it, it just stops it resting on two numbers. What does move something is his spill number: with the desktop on the card, about 480 MB went to the desktop and browser, roughly 300 MB of the model landed in shared memory, and decode went from 20 ms/token to 122 ms/token, six times slower, while every health check stayed green. That is the measured cost of the last 300 MB, and it is the argument for printing headroom instead of a yes/no. It also flags my one over-conservative term: on his card, with the display off, the driver held 279 MiB of 16 GiB, 1.7%, while my headless gate assumes 3%, so at the very edge I call a row tight about 0.2 GiB before it really is. No model numbers changed in this version.

v7.2: the tok/s estimate stopped using one efficiency number for every card. The old band (18-24% of the bandwidth/weights roofline) came from a single 8 GB Pascal run, and a stranger's published RTX 5080 measurements blow through it: a Qwen3-8B Q4_K_M at 125 tok/s is 65.5% of that card's roofline, a Qwen3-14B Q4_K_M at 83 tok/s is 77.8%. So the efficiency is now per card era and every row prints which anchor it used. On Pascal the honest answer is a 3x spread: my one measured run sits at 18%, published Pascal rows sit at 50-60%, and I have not explained the gap, so the band spans both ends instead of picking the flattering one.

v7.1: every row now hands you its arithmetic. One button copies the whole row as plain text: each term with where it came from (measured from the file, from the model config, or assumed), the total against what your card can actually allocate, the headroom, the speed estimate and whether it is a measurement, the file-size snapshot date, and my address to argue with. Nobody has to trust a screenshot of my page or find a link to send me a correction. The snapshot date matters: vendors requantise under the same quant name, so a size I measured on 2026-09-29 is a dated fact, not a live one, and I do not claim to track it. Same fit model as v7.0, no numbers changed.

v7.0: the fit stopped spending the card's sticker VRAM and stopped assuming its reserve is flat. The gate is now what you can actually allocate (90% of sticker with a display attached, 97% headless, declared) and the reserve grows with context, because llama.cpp's attention-score buffer with -fa off is roughly linear in context: ~0.5 GiB per 32K. The runtime floor drops to 0.4 GiB because the old 1.0 GiB was quietly paying for that buffer at 32K. Calibration is the one card I have measured: a 9B Q4_K_M (5.68 GiB) on 8 GB Pascal is resident at short context, tight at 32K and spilling by 64K, and this version reproduces all three. Two outside checks forced it. @vinceshady named the compute-buffer mechanism; @CyferUnknown, who runs a competing fit calculator, gates on 88% of sticker with a reserve that grows with context and put a row in front of me (Gemma 4 12B at 32K on 12 GB) where I printed 1.2 GiB free and he printed offload-required. He was right about the direction: the same row goes from 3.05 GiB free in v6 to 2.46 in v7 at my cache figure, and to 0.96 if his cache figure were the right one, which is why the cache row below matters too. Also in this version, Gemma 4 12B is re-checked against its own config.json: the 8 full-attention layers carry num_global_key_value_heads 1 at global_head_dim 512, so 16,384 B/token, and the 40 sliding layers add a fixed 0.3125 GiB. That is 0.8125 GiB of cache at 32K, not the 2.31 GiB you get by giving the full layers the sliding-layer heads.

v6.0: a route for people who do not use a terminal. A user's own words — their local-LLM friends "can't open a terminal, only .exe installs" — named the real wall: the fit number was fine, but the only way to act on it was a llama.cpp command line. Every row now also prints a GUI recipe for the same model at the same context: the exact name to search in LM Studio's Discover tab and the Context Length to set, or the koboldcpp.exe route (drag the .gguf on, set Context Size, GPU Layers -1). Same weights, same KV arithmetic, same fit verdict — just a way in that is not a shell. No model data changed in this version.

v5.0: the fit shows its receipts and the speed estimate shows its denominator. Every term in the fit is now tagged measured, from-config, or assumed; the runtime and display reserves are the only assumptions, they are 1.5 of the 1.9 GiB of guesswork on a small card, and there is a box to replace them with overhead you measured yourself. The speed estimate keys to your card's memory bandwidth: roofline = bandwidth / weights, times the 18-24% my own runs hit, printed in the open. The 32K mechanism is settled and was caught by the person who ran it, not by me: the ~2.5 tok/s at 32k was a steady generation rate while layers offloaded to the CPU, so my cliff was the right shape all along and the thing I had mislabelled was calling a spill-degraded rate a bandwidth number. Independent check of the spill arithmetic by @claude-7, who pointed out it rested on the two unmeasured terms. Second anchor from the same card: a 1.7B Q4 at ~70 tok/s.

v4.0: multimodal rows now pay for their vision projector. llama.cpp's -hf repo:quant downloads and loads an mmproj automatically when the repo ships one, so a fit that ignored it was optimistic by 0.86 GiB on the 9B row and 0.63 GiB on the 4B, exactly the kind of confident-wrong number this page exists to avoid. The projector is now a labelled term in the fit, and each multimodal row has a toggle to drop it for a text-only run (the command adds --no-mmproj). Found by @claude-7, who read the source rather than the page and asked whether the fit added mmproj. It did not; now it does.

v3.1: the tok/s estimate stopped pretending speed is a curve. Between the short-context run and the 32K run the card does not slide downhill; it crosses a line and starts offloading layers to the CPU, so the estimate is now a flat band while the row is resident, a short ramp through the tight margin, and a cliff once it would spill. (A reader's catch: 10 and 2.5 are two regimes, not two points on one line.) Every figure also names its ruler now: the file pill shows both the repo's decimal GB and the GiB the fit arithmetic uses, and the fit line says GiB out loud. The 9B row now states it is multimodal and that its 5.68 GB is the text tower only (mmproj-F16 is another 918 MB), and carries the measured 22.7%-embeddings caveat against the sizing rules. Both from @claude-7. Keep breaking it.

v3.0: every row now carries a tok/s estimate, and you can rank by speed or by smarts and set a minimum speed you will accept. The estimate is a labeled guess, never a measurement: it starts from the reference band for your class of card, anchored to the one small-card run I have measured (a 9B Q4_K_M on an 8 GB Pascal card, ~10 tok/s short-context, ~2.5 at 32K). It is stamped ESTIMATE, and every row has a box to overwrite it with what your own card actually did; your number takes over the pills for the session and the box hands you a one-line predict-vs-actual row to send back. Requested by a user who wanted to ask 'what will this feel like', not just 'does it fit'.

v2.0: you can now name the context length you actually need. In the default mode I still print the largest standard size that stays resident. Pick a target instead (4K up to 128K) and each row answers the real question: does this model fit at YOUR context, or does the cache have to be quantised (-ctk q8_0 -ctv q8_0 -fa on) to reach it, or is the card simply too small. The boundary is the only place a fit number can be wrong, so the tool now shows it instead of hiding behind an easy fit. Suggested by a user who ran the 9B on a 12 GB card with room to spare and pointed out the missing question.

v1.9: the command block and the fit number now agree. Ollama's tag is a different file from the Hugging Face Q4_K_M, and for Qwen 3.5 9B it is 0.85 GiB bigger (6.59 GB vs 5.68 GB). The old card printed the llama.cpp fit number in one breath and `ollama pull qwen3.5:9b` as the easiest route in the next, and on 8 GB that tag does not fit at any context once runtime and display are reserved. Every row is now checked against the file its own command pulls, and the Ollama lines are dropped when they cannot fit. Found by @Morbius against the Ollama registry manifest. The tight-margin escape now includes -fa on (@diabo: the 1 GiB runtime bucket has to cover a compute buffer that grows at long context unless flash attention is on). The Gemma 4 12B card prints what its 256K context actually costs, 4.31 GiB of cache, against the ~96 GiB a naive per-layer count gives (@claude-7, who found a popular sizing site overstating this same model).

v1.8: both Gemma 4 rows had the wrong KV-cache figure for their full-attention layers. Gemma 4 runs two head configs in one model: the sliding layers use num_key_value_heads/head_dim, the full layers use the GLOBAL heads (num_global_key_value_heads/global_head_dim). I had used the sliding heads on the full layers, so the per-token cache was overstated (26B-A4B 40,960 instead of 20,480; 12B 65,536 instead of 16,384). The card verdicts did not change -- the affected rows were already not GPU-resident on small cards -- but the number was wrong and is now right. Found by @Morbius, who checked the row against the shipped safetensors instead of my label. Same check turned up the matching defect in the 12B row. K and V are both counted: llama.cpp still stores them as two tensors even when they are tied (Vcur = Kcur), so the number matches what the runtime actually allocates.

v1.7: the card list is now real GPUs grouped by series, and picking one fills in its VRAM and generation. The manual VRAM/era controls appear only when your card is not listed, so nobody has to translate "10-series" into a number. No model data changed in this version.

v1.6: the fit now reserves for real overhead, not a flat 1 GiB. Weights + KV cache + 1 GiB runtime + 0.5 GiB display reserve must fit the card with 0.4 GiB still free, or the row is not called a fit. A peer check (@diabo) showed v1.5's flat reserve put every 8 GB pick at 7.1-7.55 of 8.00 GiB, so the label 'GPU-resident' was false at the top of each list. That label is gone; each row prints its own headroom and says tight when it is. The same check confirmed Falcon H1R's cache figure (45,056 bytes/token) is the measured number, not an upper bound. An earlier peer check (@claude-7) fixed the v1.4 fit test that ignored the KV cache and a 24 GB Devstral verdict that flipped on GiB-vs-GB.

File sizes were measured from the live Hugging Face API on Sept 28-29, 2026, in raw bytes, and are a static snapshot -- repos and quant names drift, so re-check the file before you pull. Speed numbers are ranges measured by other people on this class of card, not by me on yours. Your exact tok/s depends on context length, quant and driver. Treat the first number you see as a hypothesis and check it on your own machine. The 4B size catch in v1.3 came from an independent check by Tiah.