# Gotchas — every failure hit during this build, and its cause

Ordered roughly by how much time each one costs if you don't know it.

## Build / toolchain

**CUDA 13.x crashes the Blackwell MMQ kernel.** Use **12.8**. `winget` only offers
13.3, so pull 12.8.1 directly from NVIDIA's archive. Symptom if you get it wrong:
either a segfault, or a silent fall back to cuBLAS costing 5–6× on prefill
(12.8+MMQ ~6,500 t/s vs 13.x+cuBLAS ~750 t/s in published A/B tests).

**`GGML_CUDA_FORCE_CUBLAS` persists in `CMakeCache.txt`.** It survives a
reconfigure. Delete `build/` between attempts or you silently stay on the slow
path with no error. Verify after building:
`grep FORCE_CUBLAS build/CMakeCache.txt` must read `OFF`.

**One invalid CUDA subpackage name aborts the entire silent install.**
Exit code `-522190823` (`0xE0E00019`), no useful error text, nothing installed.
There is no `cuda_cccl`; it is `nvrtc_12.8` and `nvml_dev_12.8`, *not* the
`cuda_`-prefixed forms. Names are in NVIDIA's Windows install guide — do not guess.

**Omit `Display.Driver` from the subpackage list.** The 12.8.1 installer bundles
driver 572.61. If your installed driver is newer (610.88 here), a plain `-s`
silently downgrades it. Enumerate toolkit subpackages explicitly.

**MSVC 14.44 works with CUDA 12.8** — no `-allowUnsupportedCompiler` needed,
despite 14.44 postdating the toolkit. Don't add the flag pre-emptively.

**Ninja does not auto-discover MSVC.** The Visual Studio generator does; Ninja
doesn't. Import `vcvars64.bat` into the environment before `cmake` configure or
the compiler test fails.

**The CUDA installer does not add its `bin` to the machine PATH.** llama.cpp
binaries then die with `-1073741515` = `STATUS_DLL_NOT_FOUND`, no message.
Add `...\CUDA\v12.8\bin` to PATH.

**Git Bash cannot run the built `.exe`s.** Fails on `api-ms-win-crt-heap-l1-1-0.dll`
(UCRT resolution). Run from PowerShell. Note the failure line starts with the exe
path, not `error:` — a grep for `^error` will miss it and look like success.

## Flags

**`-no-cnv` no longer exists.** Use `-st` / `--single-turn`. It is still present
in `common/arg.cpp` but not registered for this binary — check `--help` output,
not the source.

**`llama-bench` has no speculative-decoding flags.** Use `llama-cli` or
`llama-server` for any MTP measurement.

**`llama-cli` reports timing as** `[ Prompt: N t/s | Generation: N t/s ]` —
not the `llama_perf_context_print` / `eval time` format older guides parse.

**Reasoning effort: only `low`, `medium`, `xhigh` exist for this model.**
`minimal` throws `Jinja Exception: Unexpected reasoning effort minimal.`
`high` **silently falls back** to the `xhigh` default — it produces plausible
numbers that are actually just the default, which is worse than an error.

## Benchmarking method

**Fix the generation cap high enough or high-effort runs truncate.** At `-n 3072`
the `xhigh` run was still mid-thinking; its "time to answer" was really "time to
hit the cap". Even at `-n 16384` it never finished. Always verify the output ends
with a complete answer, not mid-token.

**`--temp 0` + fixed seed makes speculative decoding lossless** — every draft
width generates byte-identical output, so timings are directly comparable and
run-to-run variance is ~±0.1 t/s. If you see more spread than that, something
else is wrong.

**Speculative decode is single-stream.** Benchmark at `--parallel 1`; the gain
disappears at higher parallelism.

**The n-max curve is jagged, not monotonic.** n5 (51.7) beats n6 (47.1) beats
n3 (43.3). This is real kernel-tiling behaviour, confirmed across 3 reps — do not
assume a smooth curve and interpolate. **Sweep your own card**: the community
default for 24 GB parts is n-max 2, which would have left 5.4 t/s on the table.

## Runtime

**Windows silently spills VRAM overflow to system RAM.** Since driver 536.40 there
is no OOM — you get a 5–10× slowdown with no error, and llama.cpp still reports
"using GPU". Set NVIDIA Control Panel → Manage 3D Settings → **CUDA - Sysmem
Fallback Policy → Prefer No Sysmem Fallback**, scoped to `llama-server.exe`.
Policy binds at CUDA context creation, so restart the process. Critical at 256K
context, where only ~958 MiB of headroom remains.

**Power limit did nothing.** 95 W → 175 W moved decode 40.86 → 40.94 t/s (inside
error bars) despite the GPU genuinely drawing 171 W at 1987–2472 MHz. Decode is
memory-bandwidth-bound and GDDR7 clocks don't scale with power budget. Don't
spend time here.

**Context speeds are measured with the window ALLOCATED but EMPTY.** Allocation is
nearly free (2% from 32K to 256K). Filling it is not — published far-context
figures for this class of setup fall to ~12 t/s near 261K actually filled.
