Link CUDA runtime, cuBLAS and driver libs into the NIF - #70
Merged
Conversation
nyo16
force-pushed
the
fix/cuda-link-and-mtp-hybrid-rollback
branch
from
August 6, 2026 00:58
b01e711 to
ac56ecb
Compare
ggml-cuda.a leaves the CUDA runtime, cuBLAS/cuBLASLt and CUDA driver API
symbols unresolved, but the Linux link line only ever added
-lstdc++ -lm -lpthread. The resulting .so failed to load at runtime with:
Failed to load NIF library: undefined symbol: cuMemCreate
(cuMemCreate is a driver API symbol used by ggml-cuda's VMM pool.)
Add the CUDA libraries to LDFLAGS whenever the CUDA backend is selected,
deriving the toolkit root from nvcc's location rather than hardcoding
/usr/local/cuda. The stubs directory is included so -lcuda resolves on
build hosts without a driver installed (e.g. release CI); the real
libcuda.so.1 is picked up from the driver at load time.
Verified on 3x RTX 3090 / CUDA 13.3: LlamaCppEx.devices() now reports the
GPU with backend "CUDA" instead of silently falling back to CPU.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
On the needs_ckpt path (hybrid models such as Qwen 3.6, where partial
seq_rm is unsupported and the recurrent state is snapshot/restored), the
rollback re-decode rebuilt the KV from the last n_accepted_total entries
of `prompt`. That slice is wrong at both ends:
- it omits the token that was `sampled` at the top of the iteration,
which is batch element 0 at pos n_past and is NOT yet in the KV, so a
real token was silently dropped from the context; and
- it includes the final emitted token, which becomes the next
iteration's `sampled` and is decoded again as batch element 0 — so
that token ended up in the context twice.
Every rollback therefore deleted one token and duplicated another. The
visible symptom was degenerate output, e.g. Qwen3.6-35B-A3B emitting
"when a message arrives arrives arrives arrives ..." for an entire
generation, with an inflated acceptance rate (the draft head correctly
predicts the repetition it just created).
Start the slice one token earlier so the redo batch is
[sampled, tok_0 .. tok_{t-2}], leaving the final emitted token to be
decoded next iteration. The dense path (native partial seq_rm) was
already correct and is unchanged.
After this fix, greedy MTP output is invariant to n_draft — the property
correct speculative decoding must have — and matches plain decoding
exactly on low-entropy prompts.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The original fix was right about the symptom and wrong about the cause.
Adding -lcudart -lcublas -lcublasLt -lcuda does resolve cuMemCreate, but it
derived the toolkit root from `which nvcc` alone, and nvcc is precisely the
thing that is missing on the hosts where this breaks. DGX OS installs it via
/etc/profile.d/nv_paths.sh, which `ssh host mix compile`, systemd units and
most CI shells never source; environment modules behave the same way. Measured
in a CUDA 13.0.2 container with nvcc off PATH:
master LLAMA_BACKEND=cuda -> GGML_CUDA=ON, zero CUDA libs linked
LLAMA_BACKEND=auto -> silent CPU-only build
the previous fix LLAMA_BACKEND=cuda -> libs but no -L, cannot find -lcudart
LLAMA_BACKEND=auto -> silent CPU-only build
Discovery now happens once, above backend selection, and is shared by the
auto-detect and the link line so the two cannot disagree: CUDA_HOME, CUDA_PATH,
nvcc on PATH, /usr/local/cuda, /opt/cuda, newest /usr/local/cuda-*. It also
passes -DCMAKE_CUDA_COMPILER so cmake's own find_package(CUDAToolkit) does not
repeat the mistake, probes lib64 vs lib rather than assuming, and errors naming
CUDA_HOME when CUDA is selected with no toolkit present instead of failing later
as `cannot find -lcudart`.
Building it on real hardware then surfaced a second load failure the first fix
would have shipped: `undefined symbol: ncclAllReduce`. ggml's GGML_CUDA_NCCL
defaults ON and links libnccl through cmake whenever the build host happens to
have NCCL -- every DGX, most multi-GPU boxes. This Makefile assembles its link
line by hand from ggml's static archives, so cmake's target_link_libraries is
invisible to it. The flag is now always stated rather than inherited, off by
default because linking it makes libnccl.so.2 a load-time requirement of the
artifact; LLAMA_CUDA_NCCL=1 enables it and adds the matching -lnccl.
On the CI side, publish the CUDA builds rather than leaving every Linux user to
compile their own. x86_64-linux-gnu-cu12 and -cu13 join the existing CPU and
Metal targets. They are separate artifacts because libcudart/libcublas sonames
are major-versioned and no shim bridges them. Selection needs both a CUDA
runtime and a driver: a CUDA build links -lcuda and cannot be dlopen'd without
libcuda.so.1, so a toolkit-only machine keeps the CPU artifact instead of being
handed a NIF that fails to load. Release runners are exactly such machines,
which is why each leg states LLAMA_CUDA_VARIANT rather than trusting detection.
Nothing in CI had ever selected the CUDA backend -- the reason a NIF that could
not resolve cuMemCreate reached a release at all. A cuda-link job now builds
against both majors on every PR with the toolkit deliberately off PATH, asserts
the .so declares libcudart/libcublas/libcuda.so.1, and resolves every symbol
with ldd -r against the toolkit's driver stub, which carries the libcuda.so.1
soname and so gives a GPU-less runner a real resolution rather than a skip.
enif_* is excluded, being supplied by the BEAM at dlopen.
Verified on 2x DGX Spark (GB10, sm_121a, aarch64):
cu13, GPU loads, devices() reports CUDA/NVIDIA GB10, 31/31 layers
offloaded, smoke suite 525 tests 0 failures
cu13, no GPU link gate passes; sm_86 sm_89 sm_120a sm_121a in the fatbin
cu12, no GPU link gate passes, links libcudart.so.12/libcublas.so.12
NCCL=1 GGML_CUDA_NCCL=ON, links libnccl.so.2, loads, sees the GPU
detection toolkit without driver -> CPU; +driver -> -cu12 / -cu13
gate negative asking for cu12 against the cu13 .so fails as it should
The x86_64 legs are exercised by the new PR job on real runners; the builds
above are aarch64, which shares every code path changed here.
386 tests locally, 15 of them new, credo and actionlint clean.
The new cuda-link job did its job and then tripped over the same trap it was written to catch. Both legs built correctly on x86_64 -- the toolkit was found with nvcc off PATH, and the .so declared libcudart, libcublas and libcuda.so.1 -- but the ldd step then failed with "no libcuda stub in the toolkit". `find /usr/local -path '*/lib64/stubs/libcuda.so'` cannot match, because lib64 is a symlink to targets/<triple>/lib and find does not descend through symlinked directories. This is the third appearance of that trap in this change: it is why the artifact lookup needs `find -L`, and why the Makefile resolves the stubs directory with `$(wildcard)` instead. Use globs, which resolve symlinks, and cover both the lib64 and the targets/<triple>/lib spellings. Reproduced in a CUDA 13.0.2 container: the old find returns nothing, the glob returns /usr/local/cuda-13.0/lib64/stubs/libcuda.so.
nyo16
force-pushed
the
fix/cuda-link-and-mtp-hybrid-rollback
branch
from
August 6, 2026 03:02
33b5d79 to
683d1fd
Compare
Rebasing onto v0.8.42 turned up two changelog problems. The Unreleased notes still quoted a smoke figure measured against b10217, and the branch's MTP commit had no entry at all -- awkward now that v0.8.42 devotes a section to MTP being "completely broken and now working", which a reader would reasonably assume covers this too. It does not: that fix was `load_mtp`, the layers never reaching the GPU. This one is the KV rebuild after a partial accept re-decoding from the wrong offset, which only bites once MTP is working, and quietly. Both entries now say which is which. The benchmark section gains the CUDA numbers this branch exists to make possible. Qwen3.6-35B-A3B UD-Q4_K_XL on a DGX Spark GB10 runs at 62.1 tok/s against 43.8 on the M1 Max already in the table -- same model, same quantization, so the 1.42x is a real comparison rather than two unrelated rows. Five runs spanning 61.7 to 62.2 make the spread 0.9%, comfortably clear of the gap. The section preamble claimed everything below it was an M4 Max Metal measurement, which stopped being true the moment a CUDA table appeared; the attribution now sits with each subsection. Also recorded is how the numbers were taken, because both mistakes are easy and silent: each run uses a distinct prompt, since repeating one hits context reuse and reports throughput the engine never reached, and tokens are counted by re-encoding the output rather than by counting stream chunks, which are not tokens and can carry several under speculation.
Trying MTP on unsloth's Qwen3.6-35B-A3B-UD-Q4_K_XL returns
{:error, "failed to create context"}. The actual reason is one line above it in
llama.cpp's log -- "context type MTP requested but model doesn't contain MTP
layers" -- which the caller may not be showing at all, and which does not
resemble the error they got.
This is a third distinct MTP failure, after the n_draft guard and v0.8.42's
load_mtp guard, and unlike those no flag recovers it: the checkpoint has no MTP
head to load. It is also the one users hit first, because most GGUF conversions
of an MTP-capable model drop the head. Same model, same publisher, same
quantization: Qwen3.6-35B-A3B-UD-Q4_K_XL has zero nextn layers, while the
separate Qwen3.6-35B-A3B-MTP-GGUF build has them. Nothing in the filename says
so.
Upstream already exposes llama_model_n_layer_nextn; it just was not wired up.
`init/2` gains a cond branch beside the existing two, matching how they read:
refuse before creating any context, and name the remedy -- here, that the
publisher usually ships an MTP-preserving build as a separate -MTP repository.
Verified on a DGX Spark GB10 against the real file: nextn layers reports 0 and
init/2 returns the new message. Two tests cover it, gated `:smoke` so they use
the ordinary generation model CI already downloads -- an ordinary checkpoint
being exactly the right fixture. Deliberately not tagged `:mtp` as well, since
--include beats --exclude and a second gate tag would drag them into runs with
no model for them. 11 tests in mtp_test, 0 failures; 391 in the suite.
The README told NVIDIA users to run `n_draft: 3`, on the grounds that this is what upstream's 2x figure assumes. Measured on a DGX Spark that is the worst of the three values tried: it lands 2% *below* no speculation at all, and 4 lands 14% below. The optimum is 2. MTP is worth having here -- 61.4 -> 71.2 tok/s, +16% -- which is a real result and also nothing like 2x. Plain and MTP ran interleaved in one process so any drift hit both arms, 11 samples each. MTP is the noisier arm by an order of magnitude (21% range against 2%), but the ranges do not overlap at all: its slowest run beat plain's fastest, so the gain does not rest on the medians. The engine's own counters explain the sweep rather than leaving it as a curiosity. Going from n_draft 2 to 3 drops acceptance from 68.5% to 57.2% while buying 15% more tokens per iteration, and pays 31% more drafting plus 10% more verify for them -- the third draft position is the least likely to be accepted, so marginal acceptance decays faster than marginal cost and the extra draft loses money. That is the same shape as the Apple Silicon note directly above, for a different reason: Metal's problem is that a wide verify is expensive, while GB10 is a unified-memory part whose MoE decode is memory-bandwidth bound, so a wider verify reads more expert weights per step. Two unrelated causes, one conclusion, which is why the option doc now says the optimum is hardware-specific and worth measuring instead of naming a default that only holds on datacenter parts. Also records that the plain UD-Q4_K_XL has no MTP head and the separate -MTP build does, since that is what sends people looking for a benchmark they cannot reproduce.
nyo16
added a commit
that referenced
this pull request
Aug 6, 2026
#70's master run was cancelled before its matrix even expanded, which is a reasonable thing to do to a merge that has just added ~20 minutes of nvcc to every landing. But cancelling it leaves master with no verdict at all, so the fix is to make the job cheap to leave on rather than something worth killing. A squash merge of an up-to-date branch lands exactly the tree the pull request tested, and re-running the CUDA legs against it re-confirms a known answer. The case that genuinely needs re-testing is master moving underneath the branch -- v0.8.42 landed under #70 mid-review, so this is not hypothetical. So the job now runs on every pull request, where it is the gate, and on master only when a CUDA build input moved: Makefile, c_src/, mix.exs, mix.lock, the vendor/llama.cpp submodule, or the workflow itself. Docs and Elixir-only merges skip it. Everything unexpected resolves to running: a force-push, an absent base commit, a zero SHA. The decision is computed with git rather than a third-party paths filter, to avoid adding an action to the surface this repo takes care to pin, and it is written as plain `if` blocks because whether `set -e` exits on a `test ... && cmd` whose left side is false is a corner of the standard nobody should have to recall while editing CI. Simulated across seven cases: PR runs, docs-only push skips, Makefile / submodule / mix.exs pushes run, empty and zero base SHAs run. The workflow also gains a concurrency group, since two runs on one branch now means two twenty-minute jobs queueing behind each other. master keys on run_id so landed commits are never cancelled by a later push -- the failure mode this commit exists to fix. Also carries the changelog note corroborating #70's MTP rollback fix against upstream's reference path, which was written after that PR merged.
nyo16
added a commit
that referenced
this pull request
Aug 6, 2026
* Corroborate the MTP rollback fix against upstream's reference path
The off-by-one entry rested on reading our own code and reasoning about what
the KV should contain. llama.cpp implements the same accept step for its server,
so it can be checked rather than argued.
common_sampler_sample_and_accept_n (common/sampling.cpp) samples at batch
indices 0..k and returns every accepted token, the one at index 0 being the
token that follows the `sampled` occupying batch element 0. server-context.cpp
then commits it as:
slot.prompt.tokens.insert({ids.begin(), ids.end() - 1});
slot.sampled = ids.back();
slot.mem.seq_rm(slot.id, slot.prompt.tokens.pos_next(), -1);
Every accepted token except the last goes into the context; the last is carried
into the next iteration as `sampled` and decoded there. That is exactly the
invariant this branch restores, arrived at independently, which is about as good
a second opinion as is available without an MTP GGUF to measure acceptance on.
Worth noting the mechanisms differ: upstream trims with seq_rm because its
verify batch already wrote the accepted prefix at the right positions, while
this binding rolls the target back to n_past and re-decodes. Only the resulting
context has to agree, and now it does. Whether the re-decode is needed at all is
a separate question this does not touch.
* ci: stop paying twenty minutes to re-test a tree we just tested
#70's master run was cancelled before its matrix even expanded, which is a
reasonable thing to do to a merge that has just added ~20 minutes of nvcc to
every landing. But cancelling it leaves master with no verdict at all, so the
fix is to make the job cheap to leave on rather than something worth killing.
A squash merge of an up-to-date branch lands exactly the tree the pull request
tested, and re-running the CUDA legs against it re-confirms a known answer. The
case that genuinely needs re-testing is master moving underneath the branch --
v0.8.42 landed under #70 mid-review, so this is not hypothetical. So the job now
runs on every pull request, where it is the gate, and on master only when a CUDA
build input moved: Makefile, c_src/, mix.exs, mix.lock, the vendor/llama.cpp
submodule, or the workflow itself. Docs and Elixir-only merges skip it.
Everything unexpected resolves to running: a force-push, an absent base commit,
a zero SHA. The decision is computed with git rather than a third-party paths
filter, to avoid adding an action to the surface this repo takes care to pin,
and it is written as plain `if` blocks because whether `set -e` exits on a
`test ... && cmd` whose left side is false is a corner of the standard nobody
should have to recall while editing CI. Simulated across seven cases: PR runs,
docs-only push skips, Makefile / submodule / mix.exs pushes run, empty and zero
base SHAs run.
The workflow also gains a concurrency group, since two runs on one branch now
means two twenty-minute jobs queueing behind each other. master keys on run_id
so landed commits are never cancelled by a later push -- the failure mode this
commit exists to fix.
Also carries the changelog note corroborating #70's MTP rollback fix against
upstream's reference path, which was written after that PR merged.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ggml-cuda.a leaves the CUDA runtime, cuBLAS/cuBLASLt and CUDA driver API symbols unresolved, but the Linux link line only ever added -lstdc++ -lm -lpthread. The resulting .so failed to load at runtime with:
(cuMemCreate is a driver API symbol used by ggml-cuda's VMM pool.)
Add the CUDA libraries to LDFLAGS whenever the CUDA backend is selected, deriving the toolkit root from nvcc's location rather than hardcoding /usr/local/cuda. The stubs directory is included so -lcuda resolves on build hosts without a driver installed (e.g. release CI); the real libcuda.so.1 is picked up from the driver at load time.
Verified on 3x RTX 3090 / CUDA 13.3: LlamaCppEx.devices() now reports the GPU with backend "CUDA" instead of silently falling back to CPU.