Link CUDA runtime, cuBLAS and driver libs into the NIF

.github/workflows/ci.yml

@@ -192,6 +192,109 @@ jobs:
 
       - run: mix dialyzer
 
+  # Nothing in CI ever selected the CUDA backend, which is how a NIF that could
+  # not resolve `cuMemCreate` reached a release. Runners have no GPU, so this
+  # cannot execute a kernel; it proves the two things that were actually broken:
+  # the toolkit is found without nvcc on PATH, and the link line resolves every
+  # CUDA symbol ggml-cuda leaves undefined.
+  cuda-link:
+    name: CUDA link (cu${{ matrix.major }})
+    runs-on: ubuntu-22.04
+    needs: [setup]
+    strategy:
+      fail-fast: false
+      matrix:
+        include:
+          - toolkit: '12-9'
+            major: '12'
+          - toolkit: '13-0'
+            major: '13'
+    steps:
+      - uses: actions/checkout@v6
+        with:
+          submodules: recursive
+
+      # Before setup-beam, which installs into the tool cache this would remove.
+      - name: Free disk space for the CUDA toolkit
+        run: |
+          sudo rm -rf /usr/share/dotnet /usr/local/lib/android /opt/ghc /usr/local/share/boost
+          df -h /
+
+      - uses: erlef/setup-beam@v1
+        with:
+          otp-version: ${{ env.OTP_VERSION }}
+          elixir-version: ${{ env.ELIXIR_VERSION }}
+
+      - name: Install build dependencies
+        run: sudo apt-get update && sudo apt-get install -y cmake libgomp1
+
+      - name: Install the CUDA toolkit
+        run: |
+          curl -fsSLO https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-keyring_1.1-1_all.deb
+          sudo dpkg -i cuda-keyring_1.1-1_all.deb
+          sudo apt-get update
+          sudo apt-get install -y --no-install-recommends cuda-toolkit-${{ matrix.toolkit }}
+          # Deliberately NOT added to PATH. A toolkit that is installed but not
+          # on PATH is the normal case on real machines (DGX OS ships nvcc via
+          # /etc/profile.d, which no non-login shell sources) and it is the case
+          # that used to produce a silent CPU-only build.
+          ls -d /usr/local/cuda-*
+
+      - run: mix deps.get
+
+      - name: Build the NIF against CUDA
+        run: mix compile
+        env:
+          LLAMA_BACKEND: cuda
+          LLAMA_PORTABLE: '1'
+          # One architecture. This job is about finding the toolkit and
+          # resolving the link; compiling the full release architecture list
+          # would add an hour and prove nothing more.
+          LLAMA_CMAKE_ARGS: -DCMAKE_CUDA_ARCHITECTURES=86-real
+
+      - name: The NIF must declare and resolve its CUDA dependencies
+        run: |
+          set -eu
+          # -L is required: mix makes _build/<env>/lib/<app>/priv a symlink to
+          # the project's priv/, and find does not descend into symlinked
+          # directories, so a plain find reports nothing here.
+          so=$(find -L _build -name llama_cpp_ex_nif.so | head -1)
+          test -n "$so" || { echo "::error::no NIF was built"; exit 1; }
+          readelf -d "$so" | grep NEEDED
+
+          for lib in libcudart.so.${{ matrix.major }} libcublas.so.${{ matrix.major }} libcuda.so.1; do
+            readelf -d "$so" | grep -q "\[$lib\]" \
+              || { echo "::error::$so does not link $lib"; exit 1; }
+          done
+
+          # Full symbol resolution, which is the check that catches this class of
+          # bug. `ldd -r` needs libcuda.so.1 and a runner has no driver to supply
+          # it -- but the toolkit stub exports exactly the driver API under that
+          # same soname, so pointing the loader at it resolves the real symbols
+          # rather than faking them.
+          # Globs, not `find`: `lib64` is a symlink to targets/<triple>/lib and
+          # find will not descend through it, which is why a `-path '*/lib64/*'`
+          # search comes back empty on a perfectly good toolkit.
+          stub=$(ls -1 /usr/local/cuda*/lib64/stubs/libcuda.so \
+                       /usr/local/cuda*/targets/*/lib/stubs/libcuda.so \
+                       2>/dev/null | head -1)
+          test -n "$stub" || { echo "::error::no libcuda stub in the toolkit"; exit 1; }
+          echo "driver stub: $stub"
+          mkdir -p /tmp/driver
+          ln -sf "$stub" /tmp/driver/libcuda.so.1
+          LD_LIBRARY_PATH=/tmp/driver ldd -r "$so" > /tmp/ldd.txt 2>&1 || true
+          cat /tmp/ldd.txt
+
+          # enif_* is the NIF API. The BEAM exports it from the running emulator
+          # and resolves it when it dlopens the library, so it is undefined here
+          # by design and is the one thing this gate must ignore. Everything else
+          # unresolved is a library the link line forgot -- how `cuMemCreate` and
+          # then `ncclAllReduce` each shipped a NIF that could not load.
+          if grep -iE 'undefined symbol|not found' /tmp/ldd.txt | grep -v 'undefined symbol: enif_'; then
+            echo "::error::the NIF has unresolved non-BEAM symbols; see above"
+            exit 1
+          fi
+
   inference:
     name: Inference smoke
     runs-on: ubuntu-22.04

.github/workflows/precompile.yml

@@ -47,11 +47,18 @@ jobs:
       fail-fast: false
       matrix:
         include:
+          # `toolkit` and `variant` are spelled out on every leg, empty where
+          # there is no CUDA. An absent matrix key is null, and `null != ''` in a
+          # GitHub expression resolves through numeric coercion rather than
+          # string comparison; being explicit keeps the `if:` guards below from
+          # depending on that.
           - os: macos-14
             target: aarch64-apple-darwin
             otp: '27.0'
             elixir: '1.18'
             backend: metal
+            toolkit: ''
+            variant: ''
           # OTP 25 reports NIF 2.16, OTP 26/27/28 all report 2.17, and OTP 29 is
           # the first release to report 2.18 (verified against
           # erts/emulator/beam/erl_nif.h in the OTP source). So OTP 27 and OTP 29
@@ -62,16 +69,55 @@ jobs:
             otp: '29.0'
             elixir: '1.20'
             backend: metal
+            toolkit: ''
+            variant: ''
           - os: ubuntu-22.04
             target: x86_64-linux-gnu
             otp: '27.0'
             elixir: '1.18'
             backend: cpu
+            toolkit: ''
+            variant: ''
           - os: ubuntu-22.04
             target: x86_64-linux-gnu
             otp: '29.0'
             elixir: '1.20'
             backend: cpu
+            toolkit: ''
+            variant: ''
+          # CUDA artifacts are per major version because the NIF links
+          # libcudart/libcublas/libcublasLt dynamically and those sonames are
+          # major-versioned: one Linux CUDA build cannot serve both. 22.04 is
+          # kept for the same reason as the CPU legs -- it is the oldest glibc
+          # these artifacts have to load against.
+          - os: ubuntu-22.04
+            target: x86_64-linux-gnu-cu12
+            otp: '27.0'
+            elixir: '1.18'
+            backend: cuda
+            toolkit: '12-9'
+            variant: cu12
+          - os: ubuntu-22.04
+            target: x86_64-linux-gnu-cu12
+            otp: '29.0'
+            elixir: '1.20'
+            backend: cuda
+            toolkit: '12-9'
+            variant: cu12
+          - os: ubuntu-22.04
+            target: x86_64-linux-gnu-cu13
+            otp: '27.0'
+            elixir: '1.18'
+            backend: cuda
+            toolkit: '13-0'
+            variant: cu13
+          - os: ubuntu-22.04
+            target: x86_64-linux-gnu-cu13
+            otp: '29.0'
+            elixir: '1.20'
+            backend: cuda
+            toolkit: '13-0'
+            variant: cu13
 
     runs-on: ${{ matrix.os }}
     permissions:
@@ -80,13 +126,30 @@ jobs:
       - uses: actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6
         with:
           submodules: recursive
+      # Before setup-beam, which installs into the tool cache this would remove.
+      - name: Free disk space for the CUDA toolkit
+        if: matrix.toolkit != ''
+        run: |
+          sudo rm -rf /usr/share/dotnet /usr/local/lib/android /opt/ghc /usr/local/share/boost
+          df -h /
       - uses: erlef/setup-beam@54075bcc5e249e4758d363f27d099f55d843f124 # v1
         with:
           otp-version: ${{ matrix.otp }}
           elixir-version: ${{ matrix.elixir }}
       - name: Install cmake (Linux)
         if: runner.os == 'Linux'
         run: sudo apt-get update && sudo apt-get install -y cmake
+      - name: Install the CUDA toolkit
+        if: matrix.toolkit != ''
+        run: |
+          curl -fsSLO https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-keyring_1.1-1_all.deb
+          sudo dpkg -i cuda-keyring_1.1-1_all.deb
+          sudo apt-get update
+          sudo apt-get install -y --no-install-recommends cuda-toolkit-${{ matrix.toolkit }}
+          # Not added to PATH on purpose: the Makefile resolves the toolkit
+          # itself, and leaving nvcc off PATH keeps this leg honest about the
+          # discovery path most real machines take.
+          ls -d /usr/local/cuda-*
       - name: Set version from tag
         run: |
           TAG_VERSION="${GITHUB_REF#refs/tags/v}"
@@ -125,6 +188,24 @@ jobs:
           # artifact to whatever CPU the runner happened to have and hand users a
           # SIGILL on older hardware.
           LLAMA_PORTABLE: '1'
+          # Names the artifact. Detection cannot be trusted here: it requires a
+          # driver before it will claim a CUDA target, and a release runner has
+          # a toolkit and no driver, so an unset variant would silently publish
+          # a CUDA build under the CPU name. Empty on the CPU legs, which pins
+          # them to the CPU name even on a runner that happens to have CUDA.
+          LLAMA_CUDA_VARIANT: ${{ matrix.variant }}
+      - name: The artifact must be named for the target this leg builds
+        run: |
+          set -eu
+          # A mismatch here means the matrix and the precompiler disagree about
+          # what this leg produced. Left unchecked it surfaces as the `checksum`
+          # job failing on a missing artifact, long after the build that could
+          # have explained it.
+          ls cache/
+          # Anchored on the version, so it cannot pass on a neighbouring name:
+          # `*-x86_64-linux-gnu-*` alone would happily match the cu12 artifact.
+          ls cache/*-${{ matrix.target }}-"${TAG_VERSION}".tar.gz >/dev/null \
+            || { echo "::error::this leg produced no artifact named for ${{ matrix.target }}"; exit 1; }
       - name: Upload artifacts to the draft release
         env:
           GH_TOKEN: ${{ github.token }}

CHANGELOG.md

@@ -1,5 +1,102 @@
 # Changelog
 
+## Unreleased
+
+CUDA is a first-class target: the NIF now links correctly against it, and the
+release publishes prebuilt CUDA artifacts for CUDA 12 and CUDA 13.
+
+Verified on 2x NVIDIA DGX Spark (GB10, `sm_121a`, aarch64, CUDA 13.0.2) against
+this base (llama.cpp b10280) — a source build with `LLAMA_BACKEND=cuda` loads,
+reports `backend: "CUDA"` from `LlamaCppEx.devices()`, offloads 31/31 layers to
+the GPU, and passes the smoke suite: **528 tests, 0 failures**.
+
+### Fixed
+
+- **MTP hybrid rollback corrupted the KV cache after a partial accept.** This is
+  a different bug from the `load_mtp` one fixed in v0.8.42, and the two are
+  complementary: that one stopped the MTP layers being read off disk at all,
+  this one silently misplaces context once they are working.
+
+  The verification batch is `[sampled, drafts...]` at positions starting at
+  `n_past`, so `sampled` occupies batch element 0. When only some drafts are
+  accepted the target's KV is rolled back to `n_past` and re-decoded — but the
+  re-decode started at the first *accepted* token rather than at `sampled`.
+  Every token therefore landed one position early, and the last accepted token
+  was written into the context even though it becomes the next iteration's
+  `sampled` and is decoded again there, duplicating it. Reading the slice from
+  `prompt[size - n_accepted_total - 1]` restores `sampled, accepted[0..k-2]`
+  across `[n_past, n_past + k)`, which is exactly the span `n_past` then
+  advances over.
+
+  Only reachable on a partial accept, which is why a working MTP setup can still
+  post plausible acceptance rates while quietly drifting.
+- **The CUDA NIF could not be loaded** — `ggml-cuda.a` leaves the CUDA runtime,
+  cuBLAS/cuBLASLt and the CUDA driver API unresolved, but the Linux link line
+  only ever added `-lstdc++ -lm -lpthread`. The resulting `.so` linked and then
+  died at load with `undefined symbol: cuMemCreate`, a driver-API symbol
+  ggml-cuda's VMM pool calls.
+- **The toolkit was only ever looked for on `PATH`** — and that is the one place
+  it frequently is not. DGX OS installs `nvcc` via `/etc/profile.d/nv_paths.sh`,
+  which `ssh host mix compile`, systemd units and most CI shells never source;
+  environment modules behave the same way. The two consequences were both
+  silent: `LLAMA_BACKEND=auto` produced a CPU-only build on a machine with a
+  complete toolkit, and `LLAMA_BACKEND=cuda` produced a link line with no `-L`
+  at all. Discovery now tries `CUDA_HOME`, `CUDA_PATH`, `nvcc` on `PATH`,
+  `/usr/local/cuda`, `/opt/cuda`, then the newest `/usr/local/cuda-*`, is shared
+  between backend auto-detection and the link line so they cannot disagree, and
+  passes `-DCMAKE_CUDA_COMPILER` so cmake's own `find_package(CUDAToolkit)`
+  does not repeat the mistake. Selecting CUDA with no toolkit present is now an
+  error naming `CUDA_HOME`, not `cannot find -lcudart`.
+- **`undefined symbol: ncclAllReduce`** — ggml's `GGML_CUDA_NCCL` defaults to ON
+  and links libnccl through cmake whenever the build host happens to have NCCL,
+  which every DGX and most multi-GPU boxes do. The Makefile assembles its link
+  line by hand from ggml's static archives, so cmake's `target_link_libraries`
+  is invisible to it and the symbols went unresolved. The flag is now always
+  stated explicitly rather than inherited, defaulting to OFF; `LLAMA_CUDA_NCCL=1`
+  turns it on and adds the matching `-lnccl`.
+- **Library path assumed `lib64`** — Debian's packaged `nvidia-cuda-toolkit`
+  only has `lib`, and the stubs directory is now probed rather than assumed.
+
+### Added
+
+- **`LlamaCppEx.MTP.init/2` now says when a checkpoint has no MTP head**, via a
+  new `LlamaCppEx.NIF.model_n_layer_nextn/1` wrapping upstream's
+  `llama_model_n_layer_nextn`. Previously this surfaced as
+  `{:error, "failed to create context"}`, with the real reason — llama.cpp's
+  `context type MTP requested but model doesn't contain MTP layers` — buried in
+  engine output the caller may not be showing.
+
+  This is not the `load_mtp` case and no flag recovers it: most GGUF conversions
+  of an MTP-capable model simply drop the head. Unsloth's
+  `Qwen3.6-35B-A3B-UD-Q4_K_XL` reports zero nextn layers; their separate
+  `Qwen3.6-35B-A3B-MTP-GGUF` build of the same model carries them. The message
+  now says so and points at the `-MTP` build.
+
+- **Precompiled CUDA artifacts** — `x86_64-linux-gnu-cu12` and
+  `x86_64-linux-gnu-cu13` join the existing `aarch64-apple-darwin` (Metal) and
+  `x86_64-linux-gnu` (CPU) targets, at NIF 2.17 and 2.18. `mix compile` on an
+  x86_64 Linux box with a driver and a CUDA runtime now downloads a GPU build
+  instead of silently installing a CPU one.
+
+  They are separate artifacts because the NIF links `libcudart`/`libcublas`/
+  `libcublasLt` dynamically and those sonames are major-versioned; one "Linux
+  CUDA" binary cannot serve both. Selection requires **both** a CUDA runtime and
+  a driver (`libcuda.so.1`) — a CUDA build links `-lcuda` and cannot be
+  `dlopen`ed at all without one, so a toolkit-only machine keeps the CPU
+  artifact rather than getting a NIF that fails to load. `LLAMA_CUDA_VARIANT`
+  (`cu12`, `cu13`, `none`) overrides the probe.
+- **`LLAMA_CUDA_NCCL`** build variable, off by default. See above.
+- **A CUDA link gate in CI** — nothing in CI had ever selected the CUDA backend,
+  which is how a NIF that could not resolve `cuMemCreate` reached a release. The
+  new `cuda-link` job builds against both CUDA 12 and CUDA 13 on every pull
+  request with the toolkit deliberately off `PATH`, asserts the `.so` declares
+  `libcudart`, `libcublas` and `libcuda.so.1`, and then resolves every symbol
+  with `ldd -r` — pointing the loader at the toolkit's driver stub, which
+  carries the `libcuda.so.1` soname, so a GPU-less runner can still perform a
+  real resolution. `enif_*` is excluded, being supplied by the BEAM at load.
+- **Precompiler unit tests** — `test/precompiler_test.exs` pins the artifact
+  selection rules, including the case that motivates the driver check.
+
 ## v0.8.42
 
 llama.cpp bump to b10280, on top of b10217 from v0.8.41. Unlike the last two