Bump llama.cpp to 61881b1f7 (b10280), release v0.8.42

CHANGELOG.md

@@ -1,5 +1,155 @@
 # Changelog
 
+## v0.8.42
+
+llama.cpp bump to b10280, on top of b10217 from v0.8.41. Unlike the last two
+ranges this one **does** break the upstream C API, and validating it against real
+models surfaced a separate bug that had made MTP speculative decoding inoperable
+since v0.8.41. The public Elixir API gains one option and no removals.
+
+### Fixed
+
+- **MTP speculative decoding was completely broken and is now working.** b10217
+  (in v0.8.41) added `bool load_mtp` to `llama_model_params` to stop
+  non-speculative callers paying for the MTP head's tensors (#26296, extended by
+  #26412 here). v0.8.41's notes read that field as inert because the NIF builds
+  its params from `llama_model_default_params()` and so picks up new fields
+  automatically — which is true at compile time and wrong at runtime: the default
+  is `false`, so the MTP layers stopped being read off disk. Nothing on the way in
+  objects. Both contexts build and `common_speculative_init` returns ok; the first
+  draft then fails with `verify decode failed: code=-1`, far from the cause.
+
+  `LlamaCppEx.Model.load/2` therefore gains a `:load_mtp` option (default
+  `false`, matching upstream), the flag is recorded on `%Model{}`, and
+  `LlamaCppEx.MTP.init/2` refuses a model loaded without it, naming the remedy —
+  the layers cannot be attached after the fact, so failing at `init/2` is the only
+  place the error is actionable.
+
+  This went unnoticed because the `:mtp` suite had never been run against a real
+  model: it needs an MTP-enabled GGUF, and none was available when the tests were
+  written. It is now verified end-to-end against `Qwen3.5-9B` (`qwen35` with
+  `nextn_predict_layers`), at ~67% draft acceptance with `n_draft: 3`.
+
+### Changed
+
+- **llama.cpp submodule** — Updated from ddd4ec142 to 61881b1f7 (63 commits, tag
+  b10280). Two binding-relevant signatures changed, both breaking:
+  - `llama_sampler_init_penalties` gained a leading `int32_t n_vocab` parameter
+    (#26520, which moved `n_vocab` out of `llama_sampler_data` and into the
+    penalty sampler so the CUDA backend sampler added in #25262 can size its
+    buffers). The NIF's one call site now passes
+    `llama_vocab_n_tokens(model->vocab())`. Note that the argument *count*
+    changed, so a missed call site is a compile error rather than a silent
+    reordering — but the two leading `int32_t`s are interchangeable to the
+    compiler, which is why the penalties path is now covered behaviourally
+    (a repetition-inducing prompt under greedy decoding must be diverted by
+    `penalty_repeat`, and `penalty_repeat: 1.0` must be a byte-identical no-op)
+    rather than only by constructing a sampler.
+  - `llama_sampler_init_dry` dropped its `int32_t n_ctx_train` parameter in the
+    same range. The binding does not expose the DRY sampler, so nothing to do.
+  - **Sampler semantics**: `-1` no longer means "context size" for
+    `penalty_last_n` or `dry_penalty_last_n` — history-based samplers lost their
+    full-context windows (#26524) and a negative value now clamps to `0`, i.e.
+    disabled. The NIF hardcodes `penalty_last_n = 64`, so behaviour here is
+    unchanged, but anyone who read `-1` as "whole context" upstream should note it.
+  - `include/llama.h` aside, `common/chat.h`, `common/speculative.h` and
+    `common/json-schema-to-grammar.h` are untouched in this range.
+    `common/common.h` changed only in ways the binding does not reach: the
+    `common_params_vocoder` struct was replaced by `tts_lang` / `tts_speaker_file`
+    fields on `common_params` (#26254, which also breaks the `llama-tts` binary),
+    and `common_get_env` / `common_set_env` were added.
+  - **Speculative / MTP**: `common_speculative_init` refactored its enabled-config
+    handling (#26510); MTP support added for Qwen3-Next (#25589), DeepSeek V3.2
+    (#26457), GLM-4.7-Flash (#24868) and DeepSeek V4 alongside DSpark (#25784);
+    MiMo V2 MTP tensors are loaded only when used (#26412); dflash `wo_a` reshape
+    fixed on load (#26577).
+  - **llama core / models**: tensors may be reshaped during load (#26531); the
+    indexer cache is allocated only in "full" indexer layers (#26474); MiniMax M3
+    moves MSA into a new memory implementation (#26338) and its graph no longer
+    leaves input tensors unused (#26519); a DeepSeek V4 Flash 0731 chat template
+    (#26398) and a Qwen3 specialized parser (#26252) were added.
+  - **vocab / convert**: default special token ids (#26506) and plamo2 byte tokens
+    (#26511) are now validated; `gguf-py`'s reader validates `n_dims` and guards
+    against `uint64` overflow (#25401).
+  - **ggml**: version bumped to 0.18.1 (ggml/1578) and synced; split-graph inputs
+    now use dynamic allocation (#22789).
+  - **Metal**: DeepSeek V4 Lightning Indexer (#25893) and hyper-connections
+    (#26459) implemented; `SILU_BACK` (#25982) and F16 support for binary ops
+    (#26465) added; `GGML_METAL_USE_BF16` removed from the build scripts (#26604).
+  - **CUDA**: a backend sampler for penalties (#25262); a data race fixed when
+    reusing SMEM in `block_reduce` (#26385).
+  - **SYCL**: non-contiguous concat kernel parallelized (#25852); oneDNN SDPA
+    extended to non-FP16 KV caches (#25874); iGPU classification fixed (#26105).
+  - **Vulkan / OpenCL / WebGPU**: `GATED_LINEAR_ATTN` implemented (#25601);
+    `topk_moe` fusion extended to `sqrt(softplus)` (#26124); large `q6_K` lm_head
+    routed to the flat GEMV and workgroup sizes limited for GLU (#26427, #26383);
+    f16 repeat support added to the WebGPU backend (#26307).
+  - **vendor**: BoringSSL updated twice (#26353, #26523), cpp-httplib to 0.52.0
+    (#26485), and patches applied for `subprocess.h` (#26606). None of these link
+    into the NIF, which builds with `LLAMA_BUILD_SERVER=OFF` and
+    `LLAMA_OPENSSL=OFF`.
+- **`Makefile`** — `LLAMA_COMMIT` follows the submodule to 61881b1f7. It is what a
+  Hex *source* build clones when `vendor/llama.cpp` is absent, so leaving it behind
+  gives source builds the old llama.cpp and git checkouts the new one.
+
+### Tests
+
+- **Three smoke tests asserted prefix reuse that is impossible on hybrid GDN
+  models.** `server_smoke_test.exs` (cache-scope isolation, `PromptCache.restore/4`
+  partial trim) and `llama_cpp_ex_test.exs` (the hybrid `cache_prompt` regression
+  test) all required `prefix_cache_tokens > 0`. Recurrent state cannot be rolled
+  back to an arbitrary position, so `llama_memory_seq_rm` refuses a partial range
+  and the Server *correctly* declines reuse (`server.ex:812`) — meaning the
+  hybrid-model regression test contradicted the very fallback it documents. They
+  passed only because CI's generation model is a dense one.
+
+  Each now branches on `LlamaCppEx.TestModels.seq_rm_kind/1` (a memoised probe of
+  `common_context_can_seq_rm`) and asserts the documented behaviour for the model
+  in hand: reuse on `:part`, the full-reset fallback and `{:error, :seq_rm_refused}`
+  on `:full`. No branch is a skip, and neither `case` has a catch-all, so an
+  unexpected verdict raises instead of passing quietly. Verified both ways —
+  502 passed against a `:part` model and 502 against a `:full` one.
+
+  This was **not** a regression from the bump: `llama-memory-recurrent.cpp` and
+  both hybrid memory implementations are byte-identical between b10217 and b10280.
+
+### Known issues
+
+- **Cancelling an MTP stream and immediately reusing the session aborts the VM.**
+  `request_cancel` sets a flag and the MTP loop then stops without emitting a
+  terminal event (`llama_nif.cpp:2009`), so `Generator.stop/1` has no completion
+  signal to await and returns while the dirty scheduler may still be inside
+  `llama_decode`. For `LlamaCppEx.stream/3` this is harmless — each call owns a
+  context that dies with it — but an `%MTP{}` session holds two long-lived contexts
+  that every call shares, so the next `generate/3` can put a second writer on a KV
+  cache the cancelled loop has not released, tripping
+  `GGML_ASSERT(offset + size <= ggml_nbytes(tensor))` or segfaulting.
+
+  Pre-existing and unrelated to this bump; it became reachable only because MTP
+  works again. Fixing it means acknowledging cancellation in the NIF and in the
+  cancellation protocol that non-MTP streaming shares, so it is deliberately not
+  bundled here. `LlamaCppEx.MTP`'s docs carry the warning and the reproduction
+  lives in `LlamaCppEx.MTPCancelTest`, gated behind its own `:mtp_cancel` tag —
+  its own module, without `:mtp`, because `--include` beats `--exclude` and a
+  second gate tag would abort `--include mtp` runs.
+
+### Validation
+
+Built from source against the bumped submodule (Metal, `LLAMA_BACKEND=auto`) and
+run against real GGUF models from local storage:
+
+- **374 passed** in the default suite (no model required).
+- **502 passed** with `--include smoke --include embeddings --include slow`,
+  twice: once with `Llama-3.2-3B-Instruct` (`:part` seq_rm) and once with
+  `Qwen3.5-0.8B` (`:full`, hybrid GDN).
+- **384 passed** with `--include mtp` against `Qwen3.5-9B` (MTP head).
+- Generation, chat templating, streaming, JSON-schema-constrained sampling,
+  grammar-with-penalties, and the penalties path itself were additionally checked
+  across nine architectures — `llama`, `qwen3`, `qwen35`, `qwen35moe`, `gemma4`,
+  `gpt-oss`, `bert` (Ministral), plus `Qwen3-Embedding-0.6B` for the embedding
+  paths (L2 norm exactly 1.0; paraphrase similarity 0.73 against 0.20 for
+  unrelated text).
+
 ## v0.8.41
 
 Maintenance release: llama.cpp bump to b10217, on top of b10178 from v0.8.40.

Makefile

@@ -22,7 +22,7 @@ endif
 # Pinned llama.cpp commit, used when vendor/llama.cpp has to be cloned. MUST
 # match the vendor/llama.cpp submodule; bump both together, see
 # docs/release-guide.md. Override to build the NIF against another revision.
-LLAMA_COMMIT ?= ddd4ec1428a6201e18975ea52b07c71e0f9aef26
+LLAMA_COMMIT ?= 61881b1f7f0b13d9e46d561fc25afcd6bbaec479
 
 # The commit actually on disk. A submodule can be bumped without LLAMA_COMMIT
 # following it, and the build has to key off what is really there.

README.md

@@ -539,7 +539,11 @@ Upstream llama.cpp implements more speculative types behind the same `common_spe
 - [`ggml-org/Qwen3.6-27B-MTP-GGUF`](https://huggingface.co/ggml-org/Qwen3.6-27B-MTP-GGUF)
 - [`unsloth/Qwen3.6-35B-A3B-MTP-GGUF`](https://huggingface.co/unsloth/Qwen3.6-35B-A3B-MTP-GGUF)
 
-A regular (non-MTP) Qwen 3.6 quant will fail at `LlamaCppEx.MTP.init/2` — the GGUF must contain `mtp-*` tensors.
+A regular (non-MTP) Qwen 3.6 quant will fail at `LlamaCppEx.MTP.init/2` — the GGUF must contain the MTP head's tensors. To check a file before loading it, look for a `*.nextn_predict_layers` key and `blk.N.nextn.*` tensors in its metadata.
+
+The model must also be loaded with `load_mtp: true` (see below). Upstream gates those tensors behind a load-time flag that defaults to off, and they cannot be attached afterwards, so `MTP.init/2` refuses a model loaded without it rather than letting the omission surface later as `verify decode failed: code=-1`.
+
+> **Do not reuse a session straight after abandoning a stream.** Cancellation is asynchronous and unacknowledged, and a session's two contexts are shared by every call on it, so starting the next `generate/3` immediately can put a second writer on a KV cache the cancelled draft loop has not released — which aborts the VM. Let an abandoned stream reach a terminal event, or build a fresh session. Tracked in the v0.8.42 changelog.
 
 ### Usage
 
@@ -551,7 +555,11 @@ A regular (non-MTP) Qwen 3.6 quant will fail at `LlamaCppEx.MTP.init/2` — the
 {:ok, model} =
   LlamaCppEx.load_model(
     Path.expand("~/Downloads/Qwen3.6-35B-A3B-MTP-Q4_K_M.gguf"),
-    n_gpu_layers: 999
+    n_gpu_layers: 999,
+    # Required. Upstream defaults this off so non-speculative callers do not pay
+    # for the MTP head's tensors, and they cannot be attached to an
+    # already-loaded model, so MTP.init/2 refuses a model loaded without it.
+    load_mtp: true
   )
 
 # Build the speculative session once — it owns a target context and a

c_src/llama_cpp_ex/llama_nif.cpp

@@ -325,7 +325,8 @@ FINE_NIF(device_list, ERL_NIF_DIRTY_JOB_IO_BOUND);
 std::variant<fine::Ok<fine::ResourcePtr<LlamaModel>>, fine::Error<std::string>>
 model_load(ErlNifEnv* env, std::string path, int64_t n_gpu_layers, bool use_mmap,
            int64_t main_gpu, int64_t split_mode, std::vector<double> tensor_split,
-           bool use_mlock, bool use_direct_io, bool vocab_only, bool check_tensors) {
+           bool use_mlock, bool use_direct_io, bool vocab_only, bool check_tensors,
+           bool load_mtp) {
     auto params = llama_model_default_params();
     params.n_gpu_layers = static_cast<int32_t>(n_gpu_layers);
     params.main_gpu = static_cast<int32_t>(main_gpu);
@@ -343,6 +344,14 @@ model_load(ErlNifEnv* env, std::string path, int64_t n_gpu_layers, bool use_mmap
                                              : LLAMA_LOAD_MODE_NONE;
     params.vocab_only = vocab_only;
     params.check_tensors = check_tensors;
+    // Upstream defaults this to false so that non-speculative callers do not pay
+    // for the MTP head's tensors (#26296). It has to be set at *load* time: the
+    // layers are either read off disk here or they are absent for the model's
+    // whole lifetime, and `common_speculative_init` still succeeds without them
+    // — the failure surfaces much later as `verify decode failed: code=-1` from
+    // the first MTP draft. `LlamaCppEx.MTP.init/2` therefore refuses a model
+    // loaded without this flag rather than letting that error escape.
+    params.load_mtp = load_mtp;
 
     std::vector<float> ts_float;
     if (!tensor_split.empty()) {
@@ -652,7 +661,8 @@ sampler_init(
     // Add samplers in recommended order: penalties -> top_k -> top_p -> min_p -> temp -> dist/greedy
     if (penalty_repeat != 1.0 || penalty_freq != 0.0 || penalty_present != 0.0) {
         llama_sampler_chain_add(chain,
-            llama_sampler_init_penalties(64, static_cast<float>(penalty_repeat),
+            llama_sampler_init_penalties(llama_vocab_n_tokens(model->vocab()), 64,
+                static_cast<float>(penalty_repeat),
                 static_cast<float>(penalty_freq), static_cast<float>(penalty_present)));
     }
 

lib/llama_cpp_ex/model.ex

@@ -4,9 +4,9 @@ defmodule LlamaCppEx.Model do
   """
 
   @enforce_keys [:ref]
-  defstruct [:ref]
+  defstruct [:ref, load_mtp: false]
 
-  @type t :: %__MODULE__{ref: reference()}
+  @type t :: %__MODULE__{ref: reference(), load_mtp: boolean()}
 
   @tuning_option_keys [
     :main_gpu,
@@ -18,7 +18,7 @@ defmodule LlamaCppEx.Model do
     :check_tensors
   ]
 
-  @structural_option_keys [:n_gpu_layers, :vocab_only]
+  @structural_option_keys [:n_gpu_layers, :vocab_only, :load_mtp]
 
   @doc """
   Options that are safe for a caller to forward from user-supplied opts.
@@ -58,6 +58,11 @@ defmodule LlamaCppEx.Model do
     * `:vocab_only` - Load vocabulary and metadata only, skip weights. Defaults to `false`.
     * `:check_tensors` - Validate model tensor data on load. Defaults to `false`,
       because the check walks every tensor and costs real time on a large model.
+    * `:load_mtp` - Load the Multi-Token Prediction head's layers, for use with
+      `LlamaCppEx.MTP`. Defaults to `false`, matching upstream, so that callers
+      who are not doing speculative decoding do not pay for the extra tensors.
+      Required for `LlamaCppEx.MTP.init/2`, which refuses a model loaded without
+      it — the layers cannot be added after the fact.
 
   > #### Load mode {: .info}
   >
@@ -96,6 +101,7 @@ defmodule LlamaCppEx.Model do
     use_direct_io = Keyword.get(opts, :use_direct_io, false)
     vocab_only = Keyword.get(opts, :vocab_only, false)
     check_tensors = Keyword.get(opts, :check_tensors, false)
+    load_mtp = Keyword.get(opts, :load_mtp, false)
 
     case LlamaCppEx.NIF.model_load(
            path,
@@ -107,9 +113,10 @@ defmodule LlamaCppEx.Model do
            use_mlock,
            use_direct_io,
            vocab_only,
-           check_tensors
+           check_tensors,
+           load_mtp
          ) do
-      {:ok, ref} -> {:ok, %__MODULE__{ref: ref}}
+      {:ok, ref} -> {:ok, %__MODULE__{ref: ref, load_mtp: load_mtp}}
       {:error, _} = error -> error
     end
   end