From a50474503aaa5cf71c959bfc7860b38bd9a6f294 Mon Sep 17 00:00:00 2001 From: Niko Maroulis Date: Wed, 5 Aug 2026 21:31:57 -0400 Subject: [PATCH] Bump llama.cpp to 61881b1f7 (b10280), release v0.8.42 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Submodule ddd4ec142 -> 61881b1f7 (63 commits, tag b10217 -> b10280). Unlike the last two ranges this one breaks the upstream C API. Required API fix: - `llama_sampler_init_penalties` gained a leading `int32_t n_vocab` (#26520, which moved n_vocab out of llama_sampler_data so the CUDA backend sampler from #25262 can size its buffers). The one call site now passes `llama_vocab_n_tokens(model->vocab())`. `llama_sampler_init_dry` also dropped `n_ctx_train`, which the binding does not expose. `-1` no longer means "context size" for penalty_last_n (#26524); the NIF hardcodes 64, so behaviour is unchanged. Fixed: MTP speculative decoding was entirely broken since v0.8.41. b10217 added `bool load_mtp` to llama_model_params, defaulting to false. v0.8.41 read that field as inert because the NIF builds its params from `llama_model_default_params()` — true at compile time, wrong at runtime: the MTP layers stopped being loaded. `common_speculative_init` still succeeds, so the failure surfaced as `verify decode failed: code=-1` far from the cause. `Model.load/2` gains `:load_mtp` (default false, matching upstream), it is recorded on `%Model{}`, and `MTP.init/2` refuses a model loaded without it since the layers cannot be attached afterwards. Unnoticed because the :mtp suite had never run against a real model; now verified end-to-end at ~67% draft acceptance. Tests: three smoke tests asserted prefix reuse that hybrid GDN models cannot do. Recurrent state cannot be rolled back to an arbitrary position, so a partial seq_rm is refused and the Server correctly declines reuse — meaning the hybrid-model regression test contradicted the fallback it documents. They passed only because CI's generation model is dense. Each now branches on `TestModels.seq_rm_kind/1` and asserts the documented behaviour for the model in hand, with no catch-all clause so an unexpected verdict raises. Not a regression from this bump: the recurrent and hybrid memory implementations are byte-identical across the range. The MTP suite also moved to its own module so the model loads once — a per-test load accumulated Metal buffers until ggml-alloc aborted the VM. Known issue: cancelling an MTP stream and immediately reusing the session aborts the VM (cancellation is unacknowledged, and a session's contexts are shared). Pre-existing, reachable only now that MTP works; documented, and the reproduction is gated behind its own :mtp_cancel tag. Fixing it changes the cancellation protocol non-MTP streaming shares, so it is not bundled here. Validation: 374 passed (default), 502 passed with smoke+embeddings+slow against both a :part model (Llama-3.2-3B) and a :full one (Qwen3.5-0.8B), 384 passed with --include mtp against Qwen3.5-9B. Generation, chat templating, streaming, JSON-schema sampling, grammar-with-penalties and the penalties path itself checked across nine architectures. mix format and dialyzer clean. --- CHANGELOG.md | 150 +++++++++++++++++++++++++ Makefile | 2 +- README.md | 12 +- c_src/llama_cpp_ex/llama_nif.cpp | 14 ++- lib/llama_cpp_ex/model.ex | 17 ++- lib/llama_cpp_ex/mtp.ex | 81 +++++++++----- lib/llama_cpp_ex/nif.ex | 3 +- mix.exs | 2 +- test/llama_cpp_ex_test.exs | 32 +++++- test/mtp_model_test.exs | 181 +++++++++++++++++++++++++++++++ test/mtp_test.exs | 127 +++++----------------- test/server_smoke_test.exs | 70 +++++++++++- test/support/test_models.exs | 37 +++++++ test/test_helper.exs | 7 +- vendor/llama.cpp | 2 +- 15 files changed, 589 insertions(+), 148 deletions(-) create mode 100644 test/mtp_model_test.exs diff --git a/CHANGELOG.md b/CHANGELOG.md index 80a8506..7ed0aa8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,155 @@ # Changelog +## v0.8.42 + +llama.cpp bump to b10280, on top of b10217 from v0.8.41. Unlike the last two +ranges this one **does** break the upstream C API, and validating it against real +models surfaced a separate bug that had made MTP speculative decoding inoperable +since v0.8.41. The public Elixir API gains one option and no removals. + +### Fixed + +- **MTP speculative decoding was completely broken and is now working.** b10217 + (in v0.8.41) added `bool load_mtp` to `llama_model_params` to stop + non-speculative callers paying for the MTP head's tensors (#26296, extended by + #26412 here). v0.8.41's notes read that field as inert because the NIF builds + its params from `llama_model_default_params()` and so picks up new fields + automatically — which is true at compile time and wrong at runtime: the default + is `false`, so the MTP layers stopped being read off disk. Nothing on the way in + objects. Both contexts build and `common_speculative_init` returns ok; the first + draft then fails with `verify decode failed: code=-1`, far from the cause. + + `LlamaCppEx.Model.load/2` therefore gains a `:load_mtp` option (default + `false`, matching upstream), the flag is recorded on `%Model{}`, and + `LlamaCppEx.MTP.init/2` refuses a model loaded without it, naming the remedy — + the layers cannot be attached after the fact, so failing at `init/2` is the only + place the error is actionable. + + This went unnoticed because the `:mtp` suite had never been run against a real + model: it needs an MTP-enabled GGUF, and none was available when the tests were + written. It is now verified end-to-end against `Qwen3.5-9B` (`qwen35` with + `nextn_predict_layers`), at ~67% draft acceptance with `n_draft: 3`. + +### Changed + +- **llama.cpp submodule** — Updated from ddd4ec142 to 61881b1f7 (63 commits, tag + b10280). Two binding-relevant signatures changed, both breaking: + - `llama_sampler_init_penalties` gained a leading `int32_t n_vocab` parameter + (#26520, which moved `n_vocab` out of `llama_sampler_data` and into the + penalty sampler so the CUDA backend sampler added in #25262 can size its + buffers). The NIF's one call site now passes + `llama_vocab_n_tokens(model->vocab())`. Note that the argument *count* + changed, so a missed call site is a compile error rather than a silent + reordering — but the two leading `int32_t`s are interchangeable to the + compiler, which is why the penalties path is now covered behaviourally + (a repetition-inducing prompt under greedy decoding must be diverted by + `penalty_repeat`, and `penalty_repeat: 1.0` must be a byte-identical no-op) + rather than only by constructing a sampler. + - `llama_sampler_init_dry` dropped its `int32_t n_ctx_train` parameter in the + same range. The binding does not expose the DRY sampler, so nothing to do. + - **Sampler semantics**: `-1` no longer means "context size" for + `penalty_last_n` or `dry_penalty_last_n` — history-based samplers lost their + full-context windows (#26524) and a negative value now clamps to `0`, i.e. + disabled. The NIF hardcodes `penalty_last_n = 64`, so behaviour here is + unchanged, but anyone who read `-1` as "whole context" upstream should note it. + - `include/llama.h` aside, `common/chat.h`, `common/speculative.h` and + `common/json-schema-to-grammar.h` are untouched in this range. + `common/common.h` changed only in ways the binding does not reach: the + `common_params_vocoder` struct was replaced by `tts_lang` / `tts_speaker_file` + fields on `common_params` (#26254, which also breaks the `llama-tts` binary), + and `common_get_env` / `common_set_env` were added. + - **Speculative / MTP**: `common_speculative_init` refactored its enabled-config + handling (#26510); MTP support added for Qwen3-Next (#25589), DeepSeek V3.2 + (#26457), GLM-4.7-Flash (#24868) and DeepSeek V4 alongside DSpark (#25784); + MiMo V2 MTP tensors are loaded only when used (#26412); dflash `wo_a` reshape + fixed on load (#26577). + - **llama core / models**: tensors may be reshaped during load (#26531); the + indexer cache is allocated only in "full" indexer layers (#26474); MiniMax M3 + moves MSA into a new memory implementation (#26338) and its graph no longer + leaves input tensors unused (#26519); a DeepSeek V4 Flash 0731 chat template + (#26398) and a Qwen3 specialized parser (#26252) were added. + - **vocab / convert**: default special token ids (#26506) and plamo2 byte tokens + (#26511) are now validated; `gguf-py`'s reader validates `n_dims` and guards + against `uint64` overflow (#25401). + - **ggml**: version bumped to 0.18.1 (ggml/1578) and synced; split-graph inputs + now use dynamic allocation (#22789). + - **Metal**: DeepSeek V4 Lightning Indexer (#25893) and hyper-connections + (#26459) implemented; `SILU_BACK` (#25982) and F16 support for binary ops + (#26465) added; `GGML_METAL_USE_BF16` removed from the build scripts (#26604). + - **CUDA**: a backend sampler for penalties (#25262); a data race fixed when + reusing SMEM in `block_reduce` (#26385). + - **SYCL**: non-contiguous concat kernel parallelized (#25852); oneDNN SDPA + extended to non-FP16 KV caches (#25874); iGPU classification fixed (#26105). + - **Vulkan / OpenCL / WebGPU**: `GATED_LINEAR_ATTN` implemented (#25601); + `topk_moe` fusion extended to `sqrt(softplus)` (#26124); large `q6_K` lm_head + routed to the flat GEMV and workgroup sizes limited for GLU (#26427, #26383); + f16 repeat support added to the WebGPU backend (#26307). + - **vendor**: BoringSSL updated twice (#26353, #26523), cpp-httplib to 0.52.0 + (#26485), and patches applied for `subprocess.h` (#26606). None of these link + into the NIF, which builds with `LLAMA_BUILD_SERVER=OFF` and + `LLAMA_OPENSSL=OFF`. +- **`Makefile`** — `LLAMA_COMMIT` follows the submodule to 61881b1f7. It is what a + Hex *source* build clones when `vendor/llama.cpp` is absent, so leaving it behind + gives source builds the old llama.cpp and git checkouts the new one. + +### Tests + +- **Three smoke tests asserted prefix reuse that is impossible on hybrid GDN + models.** `server_smoke_test.exs` (cache-scope isolation, `PromptCache.restore/4` + partial trim) and `llama_cpp_ex_test.exs` (the hybrid `cache_prompt` regression + test) all required `prefix_cache_tokens > 0`. Recurrent state cannot be rolled + back to an arbitrary position, so `llama_memory_seq_rm` refuses a partial range + and the Server *correctly* declines reuse (`server.ex:812`) — meaning the + hybrid-model regression test contradicted the very fallback it documents. They + passed only because CI's generation model is a dense one. + + Each now branches on `LlamaCppEx.TestModels.seq_rm_kind/1` (a memoised probe of + `common_context_can_seq_rm`) and asserts the documented behaviour for the model + in hand: reuse on `:part`, the full-reset fallback and `{:error, :seq_rm_refused}` + on `:full`. No branch is a skip, and neither `case` has a catch-all, so an + unexpected verdict raises instead of passing quietly. Verified both ways — + 502 passed against a `:part` model and 502 against a `:full` one. + + This was **not** a regression from the bump: `llama-memory-recurrent.cpp` and + both hybrid memory implementations are byte-identical between b10217 and b10280. + +### Known issues + +- **Cancelling an MTP stream and immediately reusing the session aborts the VM.** + `request_cancel` sets a flag and the MTP loop then stops without emitting a + terminal event (`llama_nif.cpp:2009`), so `Generator.stop/1` has no completion + signal to await and returns while the dirty scheduler may still be inside + `llama_decode`. For `LlamaCppEx.stream/3` this is harmless — each call owns a + context that dies with it — but an `%MTP{}` session holds two long-lived contexts + that every call shares, so the next `generate/3` can put a second writer on a KV + cache the cancelled loop has not released, tripping + `GGML_ASSERT(offset + size <= ggml_nbytes(tensor))` or segfaulting. + + Pre-existing and unrelated to this bump; it became reachable only because MTP + works again. Fixing it means acknowledging cancellation in the NIF and in the + cancellation protocol that non-MTP streaming shares, so it is deliberately not + bundled here. `LlamaCppEx.MTP`'s docs carry the warning and the reproduction + lives in `LlamaCppEx.MTPCancelTest`, gated behind its own `:mtp_cancel` tag — + its own module, without `:mtp`, because `--include` beats `--exclude` and a + second gate tag would abort `--include mtp` runs. + +### Validation + +Built from source against the bumped submodule (Metal, `LLAMA_BACKEND=auto`) and +run against real GGUF models from local storage: + +- **374 passed** in the default suite (no model required). +- **502 passed** with `--include smoke --include embeddings --include slow`, + twice: once with `Llama-3.2-3B-Instruct` (`:part` seq_rm) and once with + `Qwen3.5-0.8B` (`:full`, hybrid GDN). +- **384 passed** with `--include mtp` against `Qwen3.5-9B` (MTP head). +- Generation, chat templating, streaming, JSON-schema-constrained sampling, + grammar-with-penalties, and the penalties path itself were additionally checked + across nine architectures — `llama`, `qwen3`, `qwen35`, `qwen35moe`, `gemma4`, + `gpt-oss`, `bert` (Ministral), plus `Qwen3-Embedding-0.6B` for the embedding + paths (L2 norm exactly 1.0; paraphrase similarity 0.73 against 0.20 for + unrelated text). + ## v0.8.41 Maintenance release: llama.cpp bump to b10217, on top of b10178 from v0.8.40. diff --git a/Makefile b/Makefile index d9d9de0..aa4ae3b 100644 --- a/Makefile +++ b/Makefile @@ -22,7 +22,7 @@ endif # Pinned llama.cpp commit, used when vendor/llama.cpp has to be cloned. MUST # match the vendor/llama.cpp submodule; bump both together, see # docs/release-guide.md. Override to build the NIF against another revision. -LLAMA_COMMIT ?= ddd4ec1428a6201e18975ea52b07c71e0f9aef26 +LLAMA_COMMIT ?= 61881b1f7f0b13d9e46d561fc25afcd6bbaec479 # The commit actually on disk. A submodule can be bumped without LLAMA_COMMIT # following it, and the build has to key off what is really there. diff --git a/README.md b/README.md index b1a83c8..e3f00cc 100644 --- a/README.md +++ b/README.md @@ -539,7 +539,11 @@ Upstream llama.cpp implements more speculative types behind the same `common_spe - [`ggml-org/Qwen3.6-27B-MTP-GGUF`](https://huggingface.co/ggml-org/Qwen3.6-27B-MTP-GGUF) - [`unsloth/Qwen3.6-35B-A3B-MTP-GGUF`](https://huggingface.co/unsloth/Qwen3.6-35B-A3B-MTP-GGUF) -A regular (non-MTP) Qwen 3.6 quant will fail at `LlamaCppEx.MTP.init/2` — the GGUF must contain `mtp-*` tensors. +A regular (non-MTP) Qwen 3.6 quant will fail at `LlamaCppEx.MTP.init/2` — the GGUF must contain the MTP head's tensors. To check a file before loading it, look for a `*.nextn_predict_layers` key and `blk.N.nextn.*` tensors in its metadata. + +The model must also be loaded with `load_mtp: true` (see below). Upstream gates those tensors behind a load-time flag that defaults to off, and they cannot be attached afterwards, so `MTP.init/2` refuses a model loaded without it rather than letting the omission surface later as `verify decode failed: code=-1`. + +> **Do not reuse a session straight after abandoning a stream.** Cancellation is asynchronous and unacknowledged, and a session's two contexts are shared by every call on it, so starting the next `generate/3` immediately can put a second writer on a KV cache the cancelled draft loop has not released — which aborts the VM. Let an abandoned stream reach a terminal event, or build a fresh session. Tracked in the v0.8.42 changelog. ### Usage @@ -551,7 +555,11 @@ A regular (non-MTP) Qwen 3.6 quant will fail at `LlamaCppEx.MTP.init/2` — the {:ok, model} = LlamaCppEx.load_model( Path.expand("~/Downloads/Qwen3.6-35B-A3B-MTP-Q4_K_M.gguf"), - n_gpu_layers: 999 + n_gpu_layers: 999, + # Required. Upstream defaults this off so non-speculative callers do not pay + # for the MTP head's tensors, and they cannot be attached to an + # already-loaded model, so MTP.init/2 refuses a model loaded without it. + load_mtp: true ) # Build the speculative session once — it owns a target context and a diff --git a/c_src/llama_cpp_ex/llama_nif.cpp b/c_src/llama_cpp_ex/llama_nif.cpp index 7fbe664..ac60bfe 100644 --- a/c_src/llama_cpp_ex/llama_nif.cpp +++ b/c_src/llama_cpp_ex/llama_nif.cpp @@ -325,7 +325,8 @@ FINE_NIF(device_list, ERL_NIF_DIRTY_JOB_IO_BOUND); std::variant>, fine::Error> model_load(ErlNifEnv* env, std::string path, int64_t n_gpu_layers, bool use_mmap, int64_t main_gpu, int64_t split_mode, std::vector tensor_split, - bool use_mlock, bool use_direct_io, bool vocab_only, bool check_tensors) { + bool use_mlock, bool use_direct_io, bool vocab_only, bool check_tensors, + bool load_mtp) { auto params = llama_model_default_params(); params.n_gpu_layers = static_cast(n_gpu_layers); params.main_gpu = static_cast(main_gpu); @@ -343,6 +344,14 @@ model_load(ErlNifEnv* env, std::string path, int64_t n_gpu_layers, bool use_mmap : LLAMA_LOAD_MODE_NONE; params.vocab_only = vocab_only; params.check_tensors = check_tensors; + // Upstream defaults this to false so that non-speculative callers do not pay + // for the MTP head's tensors (#26296). It has to be set at *load* time: the + // layers are either read off disk here or they are absent for the model's + // whole lifetime, and `common_speculative_init` still succeeds without them + // — the failure surfaces much later as `verify decode failed: code=-1` from + // the first MTP draft. `LlamaCppEx.MTP.init/2` therefore refuses a model + // loaded without this flag rather than letting that error escape. + params.load_mtp = load_mtp; std::vector ts_float; if (!tensor_split.empty()) { @@ -652,7 +661,8 @@ sampler_init( // Add samplers in recommended order: penalties -> top_k -> top_p -> min_p -> temp -> dist/greedy if (penalty_repeat != 1.0 || penalty_freq != 0.0 || penalty_present != 0.0) { llama_sampler_chain_add(chain, - llama_sampler_init_penalties(64, static_cast(penalty_repeat), + llama_sampler_init_penalties(llama_vocab_n_tokens(model->vocab()), 64, + static_cast(penalty_repeat), static_cast(penalty_freq), static_cast(penalty_present))); } diff --git a/lib/llama_cpp_ex/model.ex b/lib/llama_cpp_ex/model.ex index 0e57306..e51acc1 100644 --- a/lib/llama_cpp_ex/model.ex +++ b/lib/llama_cpp_ex/model.ex @@ -4,9 +4,9 @@ defmodule LlamaCppEx.Model do """ @enforce_keys [:ref] - defstruct [:ref] + defstruct [:ref, load_mtp: false] - @type t :: %__MODULE__{ref: reference()} + @type t :: %__MODULE__{ref: reference(), load_mtp: boolean()} @tuning_option_keys [ :main_gpu, @@ -18,7 +18,7 @@ defmodule LlamaCppEx.Model do :check_tensors ] - @structural_option_keys [:n_gpu_layers, :vocab_only] + @structural_option_keys [:n_gpu_layers, :vocab_only, :load_mtp] @doc """ Options that are safe for a caller to forward from user-supplied opts. @@ -58,6 +58,11 @@ defmodule LlamaCppEx.Model do * `:vocab_only` - Load vocabulary and metadata only, skip weights. Defaults to `false`. * `:check_tensors` - Validate model tensor data on load. Defaults to `false`, because the check walks every tensor and costs real time on a large model. + * `:load_mtp` - Load the Multi-Token Prediction head's layers, for use with + `LlamaCppEx.MTP`. Defaults to `false`, matching upstream, so that callers + who are not doing speculative decoding do not pay for the extra tensors. + Required for `LlamaCppEx.MTP.init/2`, which refuses a model loaded without + it — the layers cannot be added after the fact. > #### Load mode {: .info} > @@ -96,6 +101,7 @@ defmodule LlamaCppEx.Model do use_direct_io = Keyword.get(opts, :use_direct_io, false) vocab_only = Keyword.get(opts, :vocab_only, false) check_tensors = Keyword.get(opts, :check_tensors, false) + load_mtp = Keyword.get(opts, :load_mtp, false) case LlamaCppEx.NIF.model_load( path, @@ -107,9 +113,10 @@ defmodule LlamaCppEx.Model do use_mlock, use_direct_io, vocab_only, - check_tensors + check_tensors, + load_mtp ) do - {:ok, ref} -> {:ok, %__MODULE__{ref: ref}} + {:ok, ref} -> {:ok, %__MODULE__{ref: ref, load_mtp: load_mtp}} {:error, _} = error -> error end end diff --git a/lib/llama_cpp_ex/mtp.ex b/lib/llama_cpp_ex/mtp.ex index d52841e..8d3a941 100644 --- a/lib/llama_cpp_ex/mtp.ex +++ b/lib/llama_cpp_ex/mtp.ex @@ -10,7 +10,7 @@ defmodule LlamaCppEx.MTP do :ok = LlamaCppEx.init() {:ok, model} = LlamaCppEx.load_model("Qwen3.6-35B-A3B-MTP-Q4_K_M.gguf", - n_gpu_layers: 999) + n_gpu_layers: 999, load_mtp: true) {:ok, mtp} = LlamaCppEx.MTP.init(model, n_draft: 3, n_ctx: 8192) @@ -23,14 +23,28 @@ defmodule LlamaCppEx.MTP do IO.puts("acceptance: \#{Float.round(stats.acceptance_rate * 100, 1)}%") The model GGUF must contain MTP head layers (e.g. - `ggml-org/Qwen3.6-35B-A3B-MTP-GGUF`). Loading a non-MTP quant with `init/2` - will return `{:error, _}` from `common_speculative_init`. + `ggml-org/Qwen3.6-35B-A3B-MTP-GGUF`) — look for `*.nextn_predict_layers` in its + metadata — **and** must be loaded with `load_mtp: true`. Upstream defaults that + flag to `false` so non-speculative callers do not pay for the head's tensors, + and the layers cannot be attached afterwards, so `init/2` refuses a model + loaded without it. Upstream currently requires `n_parallel = 1` for MTP. This module reflects that — a single MTP session decodes one sequence at a time. Reuse the same `%MTP{}` value across calls to `stream/3` / `generate/3` to avoid rebuilding the contexts; KV caches are cleared on each call. + > #### Do not reuse a session straight after abandoning a stream {: .warning} + > + > Cancellation is asynchronous and unacknowledged: abandoning a `stream/3` + > early (`Enum.take/2`, `break`, an exception) sets a flag that the draft loop + > notices and then exits without reporting that it has done so. A session's two + > contexts are long-lived and shared by every call on it, so starting the next + > `generate/3` or `stream/3` immediately can put a second writer on a KV cache + > the cancelled loop has not finished with — which aborts the VM rather than + > returning an error. Let an abandoned stream run to a terminal event, or build + > a fresh session with `init/2`, before using the session again. + MTP is the only speculative type this binding exposes. Upstream llama.cpp also implements EAGLE-3, DFlash (block-diffusion drafting via a separate drafter GGUF), and n-gram self-speculation behind the same @@ -81,28 +95,45 @@ defmodule LlamaCppEx.MTP do def init(%Model{} = model, opts \\ []) do n_draft = Keyword.get(opts, :n_draft, 3) - if is_integer(n_draft) and n_draft > 0 do - base_ctx_opts = forwardable_context_opts(opts) - main_opts = Keyword.merge(base_ctx_opts, ctx_type: :default) - # Match upstream server: MTP draft context is created with n_rs_seq=0. - # The MTP impl handles state rollback internally via cached hidden - # states (pending_h / verify_h), not via recurrent-state snapshots. - draft_opts = Keyword.merge(base_ctx_opts, ctx_type: :mtp, n_rs_seq: 0) - - with {:ok, main_ctx} <- Context.create(model, main_opts), - {:ok, mtp_ctx} <- Context.create(model, draft_opts), - {:ok, spec_ref} <- - LlamaCppEx.NIF.speculative_init(main_ctx.ref, mtp_ctx.ref, n_draft) do - {:ok, - %__MODULE__{ - main_ctx: main_ctx, - mtp_ctx: mtp_ctx, - spec_ref: spec_ref, - n_draft: n_draft - }} - end - else - {:error, ":n_draft must be a positive integer"} + cond do + not (is_integer(n_draft) and n_draft > 0) -> + {:error, ":n_draft must be a positive integer"} + + not model.load_mtp -> + # Upstream gates the MTP head's tensors behind a load-time flag that + # defaults to false (#26296), and nothing downstream notices they are + # missing: both contexts build and `common_speculative_init` returns ok, + # then the first draft fails with `verify decode failed: code=-1`. The + # layers cannot be attached to an already-loaded model, so refuse here + # with the actual remedy instead of surfacing that later error. + {:error, + "model was loaded without load_mtp: true, so its MTP head layers are " <> + "absent; reload it with LlamaCppEx.load_model(path, load_mtp: true)"} + + true -> + do_init(model, opts, n_draft) + end + end + + defp do_init(model, opts, n_draft) do + base_ctx_opts = forwardable_context_opts(opts) + main_opts = Keyword.merge(base_ctx_opts, ctx_type: :default) + # Match upstream server: MTP draft context is created with n_rs_seq=0. + # The MTP impl handles state rollback internally via cached hidden + # states (pending_h / verify_h), not via recurrent-state snapshots. + draft_opts = Keyword.merge(base_ctx_opts, ctx_type: :mtp, n_rs_seq: 0) + + with {:ok, main_ctx} <- Context.create(model, main_opts), + {:ok, mtp_ctx} <- Context.create(model, draft_opts), + {:ok, spec_ref} <- + LlamaCppEx.NIF.speculative_init(main_ctx.ref, mtp_ctx.ref, n_draft) do + {:ok, + %__MODULE__{ + main_ctx: main_ctx, + mtp_ctx: mtp_ctx, + spec_ref: spec_ref, + n_draft: n_draft + }} end end diff --git a/lib/llama_cpp_ex/nif.ex b/lib/llama_cpp_ex/nif.ex index 9b89f0c..946c9cf 100644 --- a/lib/llama_cpp_ex/nif.ex +++ b/lib/llama_cpp_ex/nif.ex @@ -35,7 +35,8 @@ defmodule LlamaCppEx.NIF do _use_mlock, _use_direct_io, _vocab_only, - _check_tensors + _check_tensors, + _load_mtp ), do: :erlang.nif_error(:not_loaded) diff --git a/mix.exs b/mix.exs index 25d6e95..7f8ee00 100644 --- a/mix.exs +++ b/mix.exs @@ -37,7 +37,7 @@ end defmodule LlamaCppEx.MixProject do use Mix.Project - @version "0.8.41" + @version "0.8.42" @source_url "https://github.com/nyo16/llama_cpp_ex" def project do diff --git a/test/llama_cpp_ex_test.exs b/test/llama_cpp_ex_test.exs index 3e955ef..7a39b9d 100644 --- a/test/llama_cpp_ex_test.exs +++ b/test/llama_cpp_ex_test.exs @@ -930,10 +930,28 @@ defmodule LlamaCppExTest do {:ok, t2} = LlamaCppEx.Server.generate(server, shared <> "Pick a color.", max_tokens: 12) - # The shared prefix really was reused, so the trimming path under test ran - # rather than being skipped by a cache miss. assert_receive {:started, %{prefix_cache_tokens: reused}}, 60_000 - assert reused > 0 + + # Which path the divergent request takes is decided by the model, and both + # are under test here: + # + # :part — partial trim is supported, so the shared prefix is reused and + # the trimming path itself ran rather than being skipped by a + # cache miss. + # :full — the hybrid GDN case this test is named for. The documented fix + # is to detect `:full` at server init and fall back to a full + # slot reset, so reuse is *declined*: a non-zero count here would + # mean the fallback did not fire and the M-RoPE positional + # mismatch is reachable again. + case LlamaCppEx.TestModels.seq_rm_kind(:gen) do + :part -> + assert reused > 0 + + kind when kind in [:full, :rs] -> + assert reused == 0, + "#{kind} model reported #{reused} reused tokens; the full-reset " <> + "fallback did not fire and a partial trim was attempted anyway" + end # A small model can legitimately emit EOG immediately, so the content is # not the subject here — surviving the divergent second request is. The @@ -1034,7 +1052,13 @@ defmodule LlamaCppExTest do setup do :ok = LlamaCppEx.init() - {:ok, model} = LlamaCppEx.load_model(LlamaCppEx.TestModels.path!(:mtp), n_gpu_layers: 999) + + {:ok, model} = + LlamaCppEx.load_model(LlamaCppEx.TestModels.path!(:mtp), + n_gpu_layers: 999, + load_mtp: true + ) + {:ok, mtp} = LlamaCppEx.MTP.init(model, n_draft: 3, n_ctx: 4096) on_exit(fn -> :ok end) %{model: model, mtp: mtp} diff --git a/test/mtp_model_test.exs b/test/mtp_model_test.exs new file mode 100644 index 0000000..22f6349 --- /dev/null +++ b/test/mtp_model_test.exs @@ -0,0 +1,181 @@ +defmodule LlamaCppEx.MTPModelTest do + # The MTP tests that need a real MTP-enabled GGUF. They live in their own + # module rather than a `describe` inside `mtp_test.exs` because the model has to + # be loaded in `setup_all`, which ExUnit only allows at module level — and + # putting it there in `mtp_test.exs` would force a model load on the pure + # argument-validation tests, which are meant to run in the default suite. + # + # Loading once per module is not a tidiness preference. An MTP GGUF is large + # and each `MTP.init/2` reserves two contexts whose Metal compute buffers come + # to ~1.5 GB combined. A per-test load kept several copies alive at once — NIF + # resources are freed on GC, not on scope exit — and the third test aborted the + # VM through `GGML_ASSERT(buffer_id >= 0)` in ggml-alloc once a graph + # allocation could no longer be satisfied. Each test passed in isolation; only + # the accumulation failed. + # + # GGML_METAL_NO_RESIDENCY=1 LLAMA_BACKEND=auto \ + # LLAMA_SMOKE_MTP_MODEL=/path/to/mtp-model.gguf mix test --include mtp + # + # async: false — one GPU, and the session holds the whole model. + use ExUnit.Case, async: false + + alias LlamaCppEx.MTP + + @moduletag :mtp + @moduletag timeout: 300_000 + + # One model *and* one session for the whole module. Sharing the session is not + # just a second economy: with the model loaded once, the two contexts each + # `MTP.init/2` reserves became the thing that accumulated, and a per-test + # session still aborted the VM after enough of them. `MTP` documents a session + # as reusable across calls — it clears both KV caches on entry — and the one + # test that reads cumulative counters asserts on deltas rather than absolutes, + # so nothing here needs a private session. + setup_all do + :ok = LlamaCppEx.init() + + {:ok, model} = + LlamaCppEx.load_model(LlamaCppEx.TestModels.path!(:mtp), + n_gpu_layers: -1, + load_mtp: true + ) + + {:ok, mtp} = MTP.init(model, n_ctx: 2048, n_draft: 3) + + %{model: model, session: mtp} + end + + test "init/2 builds both contexts and defaults n_draft to 3", %{model: model} do + assert {:ok, mtp} = MTP.init(model, n_ctx: 512) + assert mtp.n_draft == 3 + assert %LlamaCppEx.Context{} = mtp.main_ctx + assert %LlamaCppEx.Context{} = mtp.mtp_ctx + assert is_reference(mtp.spec_ref) + end + + test "generate/3 returns non-empty deterministic text", %{session: mtp} do + opts = [max_tokens: 16, temp: 0.0, seed: 7] + + assert {:ok, text} = MTP.generate(mtp, "The capital of France is", opts) + assert byte_size(text) > 0 + assert String.valid?(text) + + # Speculative decoding must not change *what* is generated, only how fast. + assert {:ok, ^text} = MTP.generate(mtp, "The capital of France is", opts) + end + + test "stream/3 yields the same text generate/3 returns", %{session: mtp} do + opts = [max_tokens: 16, temp: 0.0, seed: 7] + + streamed = mtp |> MTP.stream("Count to five:", opts) |> Enum.join() + assert {:ok, ^streamed} = MTP.generate(mtp, "Count to five:", opts) + end + + test "stream_events/3 emits only documented events, ending with a terminal one", %{session: mtp} do + events = + mtp + |> MTP.stream_events("The capital of France is", max_tokens: 8, temp: 0.0) + |> Enum.to_list() + + assert events != [] + + for event <- events do + assert event_kind(event) != :undocumented, "undocumented event: #{inspect(event)}" + end + + assert event_kind(List.last(events)) == :terminal + end + + test "emit_stats_every yields stats events that stream/3 filters out", %{session: mtp} do + opts = [max_tokens: 16, temp: 0.0, emit_stats_every: 2] + + events = mtp |> MTP.stream_events("Tell me a story:", opts) |> Enum.to_list() + assert Enum.any?(events, &match?({:stats, _}, &1)) + + # stream/3 is the text-only view of the same events. + pieces = mtp |> MTP.stream("Tell me a story:", opts) |> Enum.to_list() + assert Enum.all?(pieces, &is_binary/1) + end + + test "stats/1 counts drafts and accepts after generation", %{session: mtp} do + before = MTP.stats(mtp) + assert before.n_draft == 3 + + {:ok, _} = MTP.generate(mtp, "The capital of France is", max_tokens: 16, temp: 0.0) + after_gen = MTP.stats(mtp) + + assert after_gen.iters > before.iters + assert after_gen.drafts_generated > before.drafts_generated + assert after_gen.tokens_emitted > before.tokens_emitted + assert after_gen.acceptance_rate >= 0.0 and after_gen.acceptance_rate <= 1.0 + assert after_gen.drafts_accepted <= after_gen.drafts_generated + assert %{draft: _, verify: _, sample: _, total: _} = after_gen.timing_us + end + + test "print_stats/1 returns :ok", %{session: mtp} do + assert MTP.print_stats(mtp) == :ok + end + + # The event vocabulary stream_events/3 documents. + defp event_kind({:token, id, text}) when is_integer(id) and is_binary(text), do: :token + defp event_kind({:stats, snapshot}) when is_map(snapshot), do: :stats + defp event_kind({:done, snapshot}) when is_map(snapshot), do: :terminal + defp event_kind({:eog, nil}), do: :terminal + defp event_kind({:error, _reason}), do: :terminal + defp event_kind(_), do: :undocumented +end + +defmodule LlamaCppEx.MTPCancelTest do + # KNOWN BUG, kept as an executable record rather than deleted. + # + # This test does not fail — it takes the VM down, with + # `GGML_ASSERT(offset + size <= ggml_nbytes(tensor))` or a plain SIGSEGV. So it + # lives in its own module carrying only `:mtp_cancel`, and deliberately *not* + # `:mtp`: `--include` beats `--exclude` in ExUnit, so a second gate tag would + # drag it back into `--include mtp` runs and abort them. Run it on purpose: + # + # GGML_METAL_NO_RESIDENCY=1 LLAMA_BACKEND=auto \ + # LLAMA_SMOKE_MTP_MODEL=... mix test --include mtp_cancel + # + # Cancellation is fire-and-forget: `request_cancel` sets an atomic flag and the + # MTP loop then "stops quietly" (`llama_nif.cpp:2009`) without emitting any + # terminal event. `Generator.stop/1` cancels, unlinks and drains the mailbox, + # but it has no completion signal to wait on, so it returns while the dirty + # scheduler may still be inside `llama_decode`. For `LlamaCppEx.stream/3` that + # is harmless — each call owns a context that dies with it. An `%MTP{}` session + # is not: it holds two long-lived contexts and is documented as reusable across + # calls, so the `generate/3` below can start decoding on the very contexts the + # cancelled loop has not released. Two writers, one KV cache. + # + # The fix is to acknowledge cancellation — have the loop emit a terminal event + # when it observes the flag and have `Generator.stop/1` await it — which changes + # the NIF and the cancellation protocol that non-MTP streaming also uses. That + # is why it is not bundled into a submodule bump. + use ExUnit.Case, async: false + + alias LlamaCppEx.MTP + + @moduletag :mtp_cancel + @moduletag timeout: 300_000 + + setup_all do + :ok = LlamaCppEx.init() + + {:ok, model} = + LlamaCppEx.load_model(LlamaCppEx.TestModels.path!(:mtp), + n_gpu_layers: -1, + load_mtp: true + ) + + {:ok, mtp} = MTP.init(model, n_ctx: 2048, n_draft: 3) + %{session: mtp} + end + + test "a halted stream stops generation instead of running to max_tokens", %{session: mtp} do + taken = mtp |> MTP.stream("Write an endless story:", max_tokens: 400) |> Enum.take(3) + assert length(taken) == 3 + + # The MTP value is still usable afterwards. + assert {:ok, _} = MTP.generate(mtp, "2 + 2 =", max_tokens: 4, temp: 0.0) + end +end diff --git a/test/mtp_test.exs b/test/mtp_test.exs index fcaac4f..19a2abd 100644 --- a/test/mtp_test.exs +++ b/test/mtp_test.exs @@ -30,106 +30,43 @@ defmodule LlamaCppEx.MTPTest do end end - describe "the %MTP{} struct" do - test "enforces every field, because each one is a live NIF resource" do - # A partially built MTP would hand a nil reference to the NIF. - for missing <- [:main_ctx, :mtp_ctx, :spec_ref, :n_draft] do - fields = - %{main_ctx: :ctx, mtp_ctx: :ctx, spec_ref: :ref, n_draft: 3} |> Map.delete(missing) - - assert_raise ArgumentError, ~r/#{missing}/, fn -> struct!(MTP, fields) end - end - end - end - - describe "speculative decoding against a real MTP model" do - @describetag :mtp - @moduletag timeout: 300_000 - - setup do - :ok = LlamaCppEx.init() - {:ok, model} = LlamaCppEx.load_model(LlamaCppEx.TestModels.path!(:mtp), n_gpu_layers: -1) - {:ok, mtp} = MTP.init(model, n_ctx: 2048, n_draft: 3) - %{model: model, mtp: mtp} + describe "init/2 requires a model loaded with load_mtp: true" do + # Regression test for a silent break: upstream #26296 made the MTP head's + # tensors opt-in at load time via a flag defaulting to false. Nothing on the + # way in objects — both contexts build and common_speculative_init returns + # ok — and the first draft then fails with "verify decode failed: code=-1", + # far from the cause. The guard runs before any context is created, so a nil + # ref never reaches the NIF. + test "refuses a model loaded without the flag, naming the remedy" do + assert {:error, message} = MTP.init(%LlamaCppEx.Model{ref: nil}, n_draft: 3) + assert message =~ "load_mtp: true" + assert message =~ "reload it" end - test "init/2 builds both contexts and defaults n_draft to 3", %{model: model} do - assert {:ok, mtp} = MTP.init(model, n_ctx: 512) - assert mtp.n_draft == 3 - assert %LlamaCppEx.Context{} = mtp.main_ctx - assert %LlamaCppEx.Context{} = mtp.mtp_ctx - assert is_reference(mtp.spec_ref) + test "the flag is recorded on the struct, not re-derived" do + # Model.load/2 is what sets this; a hand-built struct defaults to false so + # that the guard fails closed rather than open. + refute %LlamaCppEx.Model{ref: nil}.load_mtp end - test "generate/3 returns non-empty deterministic text", %{mtp: mtp} do - opts = [max_tokens: 16, temp: 0.0, seed: 7] - - assert {:ok, text} = MTP.generate(mtp, "The capital of France is", opts) - assert byte_size(text) > 0 - assert String.valid?(text) + test "n_draft is still validated first, so its error is not masked" do + assert MTP.init(%LlamaCppEx.Model{ref: nil, load_mtp: true}, n_draft: 0) == + {:error, ":n_draft must be a positive integer"} - # Speculative decoding must not change *what* is generated, only how fast. - assert {:ok, ^text} = MTP.generate(mtp, "The capital of France is", opts) + assert MTP.init(%LlamaCppEx.Model{ref: nil, load_mtp: false}, n_draft: 0) == + {:error, ":n_draft must be a positive integer"} end + end - test "stream/3 yields the same text generate/3 returns", %{mtp: mtp} do - opts = [max_tokens: 16, temp: 0.0, seed: 7] - - streamed = mtp |> MTP.stream("Count to five:", opts) |> Enum.join() - assert {:ok, ^streamed} = MTP.generate(mtp, "Count to five:", opts) - end - - test "stream_events/3 emits only documented events, ending with a terminal one", %{mtp: mtp} do - events = - mtp - |> MTP.stream_events("The capital of France is", max_tokens: 8, temp: 0.0) - |> Enum.to_list() - - assert events != [] + describe "the %MTP{} struct" do + test "enforces every field, because each one is a live NIF resource" do + # A partially built MTP would hand a nil reference to the NIF. + for missing <- [:main_ctx, :mtp_ctx, :spec_ref, :n_draft] do + fields = + %{main_ctx: :ctx, mtp_ctx: :ctx, spec_ref: :ref, n_draft: 3} |> Map.delete(missing) - for event <- events do - assert event_kind(event) != :undocumented, "undocumented event: #{inspect(event)}" + assert_raise ArgumentError, ~r/#{missing}/, fn -> struct!(MTP, fields) end end - - assert event_kind(List.last(events)) == :terminal - end - - test "emit_stats_every yields stats events that stream/3 filters out", %{mtp: mtp} do - opts = [max_tokens: 16, temp: 0.0, emit_stats_every: 2] - - events = mtp |> MTP.stream_events("Tell me a story:", opts) |> Enum.to_list() - assert Enum.any?(events, &match?({:stats, _}, &1)) - - # stream/3 is the text-only view of the same events. - pieces = mtp |> MTP.stream("Tell me a story:", opts) |> Enum.to_list() - assert Enum.all?(pieces, &is_binary/1) - end - - test "stats/1 counts drafts and accepts after generation", %{mtp: mtp} do - before = MTP.stats(mtp) - assert before.n_draft == 3 - - {:ok, _} = MTP.generate(mtp, "The capital of France is", max_tokens: 16, temp: 0.0) - after_gen = MTP.stats(mtp) - - assert after_gen.iters > before.iters - assert after_gen.drafts_generated > before.drafts_generated - assert after_gen.tokens_emitted > before.tokens_emitted - assert after_gen.acceptance_rate >= 0.0 and after_gen.acceptance_rate <= 1.0 - assert after_gen.drafts_accepted <= after_gen.drafts_generated - assert %{draft: _, verify: _, sample: _, total: _} = after_gen.timing_us - end - - test "print_stats/1 returns :ok", %{mtp: mtp} do - assert MTP.print_stats(mtp) == :ok - end - - test "a halted stream stops generation instead of running to max_tokens", %{mtp: mtp} do - taken = mtp |> MTP.stream("Write an endless story:", max_tokens: 400) |> Enum.take(3) - assert length(taken) == 3 - - # The MTP value is still usable afterwards. - assert {:ok, _} = MTP.generate(mtp, "2 + 2 =", max_tokens: 4, temp: 0.0) end end @@ -184,12 +121,4 @@ defmodule LlamaCppEx.MTPTest do assert {:ok, []} = Task.yield(task, 2_000) || Task.shutdown(task, :brutal_kill) end end - - # The event vocabulary stream_events/3 documents. - defp event_kind({:token, id, text}) when is_integer(id) and is_binary(text), do: :token - defp event_kind({:stats, snapshot}) when is_map(snapshot), do: :stats - defp event_kind({:done, snapshot}) when is_map(snapshot), do: :terminal - defp event_kind({:eog, nil}), do: :terminal - defp event_kind({:error, _reason}), do: :terminal - defp event_kind(_), do: :undocumented end diff --git a/test/server_smoke_test.exs b/test/server_smoke_test.exs index bda32e0..3256c0f 100644 --- a/test/server_smoke_test.exs +++ b/test/server_smoke_test.exs @@ -298,7 +298,6 @@ defmodule LlamaCppEx.ServerSmokeTest do # Same tenant, same prompt: the cache is exactly what it is for. {:ok, _} = Server.generate(server, prompt, max_tokens: 8, cache_scope: "tenant-a") {%{prefix_cache_tokens: same_scope}, _} = next_telemetry() - assert same_scope > 0 # Another tenant, byte-identical prompt: no reuse. This is the leak. {:ok, _} = Server.generate(server, prompt, max_tokens: 8, cache_scope: "tenant-b") @@ -313,7 +312,31 @@ defmodule LlamaCppEx.ServerSmokeTest do # ...and it caches for itself like any other scope. {:ok, _} = Server.generate(server, prompt, max_tokens: 8) {%{prefix_cache_tokens: default_again}, _} = next_telemetry() - assert default_again > 0 + + # The two same-scope reads above are this test's positive control: without + # them, `cross_scope == 0` is equally satisfied by a server that never + # reuses anything, and M6 would look proven while proving nothing. + # + # That control is only available where the model can reuse a prefix at all. + # Each request leaves its 8 generated tokens in the slot's KV, so a repeat of + # the same prompt is a shorter match than what the slot holds and needs a + # partial trim — which a `:full` (hybrid GDN) model refuses, making the + # Server correctly decline reuse (`server.ex:1243`). Scope isolation is + # covered architecture-independently by the `best_candidate/4` and + # `covers?/4` unit tests in `server_test.exs`; here we assert whichever + # behaviour this model actually contracts for. + case LlamaCppEx.TestModels.seq_rm_kind(:gen) do + :part -> + assert same_scope > 0 + assert default_again > 0 + + kind when kind in [:full, :rs] -> + assert same_scope == 0, + "#{kind} model reused a prefix that required a partial trim" + + assert default_again == 0, + "#{kind} model reused a prefix that required a partial trim" + end end # W-6. `stream/3` and `stream_tokens/3` truncated silently on a per-token @@ -441,11 +464,18 @@ defmodule LlamaCppEx.ServerSmokeTest do setup do {:ok, model} = LlamaCppEx.load_model(LlamaCppEx.TestModels.path!(:gen), n_gpu_layers: 0) {:ok, ctx} = Context.create(model, n_ctx: 512, n_seq_max: 2) + + # Probed before the blob is built, not after: the probe clears the + # context's KV memory and decodes two throwaway tokens to reach its + # verdict, so running it later would invalidate the state captured below. + seq_rm_kind = LlamaCppEx.NIF.context_can_seq_rm(ctx.ref) + :ok = LlamaCppEx.NIF.memory_clear(ctx.ref) + {:ok, tokens} = Tokenizer.encode(model, "The capital of France is") :ok = LlamaCppEx.NIF.decode(ctx.ref, tokens) {:ok, blob} = LlamaCppEx.NIF.state_seq_get_data(ctx.ref, 0) - %{ctx: ctx, tokens: tokens, blob: blob} + %{ctx: ctx, tokens: tokens, blob: blob, seq_rm_kind: seq_rm_kind} end test "restores a full entry and reports the whole prefix", %{ @@ -469,7 +499,8 @@ defmodule LlamaCppEx.ServerSmokeTest do test "trims the unusable tail when only a prefix is reusable", %{ ctx: ctx, tokens: tokens, - blob: blob + blob: blob, + seq_rm_kind: seq_rm_kind } do entry = %{ tokens: tokens, @@ -481,8 +512,35 @@ defmodule LlamaCppEx.ServerSmokeTest do keep = entry.len - 2 - assert {:ok, ^keep} = PromptCache.restore(ctx.ref, 1, entry, keep) - assert LlamaCppEx.NIF.memory_seq_pos_max(ctx.ref, 1) == keep - 1 + result = PromptCache.restore(ctx.ref, 1, entry, keep) + pos_max = LlamaCppEx.NIF.memory_seq_pos_max(ctx.ref, 1) + + case seq_rm_kind do + :part -> + assert {:ok, ^keep} = result + assert pos_max == keep - 1 + + :full -> + # Hybrid GDN: recurrent state is not preserved per token, so upstream + # refuses to erase a partial tail. The blob is already in KV by then, + # so the only safe answer is to drop the whole sequence — reporting a + # prefix that is not there would have the next decode read stale + # positions as real. That fallback is the contract here, not an error. + assert {:error, :seq_rm_refused} = result + + assert pos_max == -1, + "a refused trim left KV behind, which the next decode would read as real positions" + + :rs -> + # Bounded partial rollback: whether two positions are within + # `n_rs_seq` is a context-configuration question, so both outcomes are + # legal. The invariant that must hold either way is that the sequence + # never reports a prefix it is not actually holding. + case result do + {:ok, ^keep} -> assert pos_max == keep - 1 + {:error, :seq_rm_refused} -> assert pos_max == -1 + end + end end test "a blob that cannot be restored leaves the sequence empty", %{ diff --git a/test/support/test_models.exs b/test/support/test_models.exs index 450218a..1fe964e 100644 --- a/test/support/test_models.exs +++ b/test/support/test_models.exs @@ -39,6 +39,43 @@ defmodule LlamaCppEx.TestModels do path(kind) || raise(unavailable(kind)) end + @doc """ + `seq_rm` support reported by the `kind` model: `:part` (any position range), + `:full` (whole sequence only) or `:rs` (bounded partial rollback). + + Prefix-cache reuse is not a property of the library alone — it is a property of + the model's memory module. Hybrid GDN architectures (Qwen 3.5 / 3.6, Mamba, + RWKV) keep recurrent state that cannot be rolled back to an arbitrary position, + so `llama_memory_seq_rm` refuses a partial range and the Server deliberately + disables prefix reuse for anything needing a trim (`server.ex:812`). Tests that + assert reuse *happened* are therefore only meaningful on `:part` models, and + must assert the documented fallback instead on `:full` ones. + + Probed once per suite run and memoised: it costs a model load plus a two-token + decode. + """ + @spec seq_rm_kind(kind()) :: :part | :full | :rs | :no + def seq_rm_kind(kind) when kind in @kinds do + case :persistent_term.get({__MODULE__, :seq_rm_kind, kind}, :miss) do + :miss -> + probed = probe_seq_rm_kind(kind) + :persistent_term.put({__MODULE__, :seq_rm_kind, kind}, probed) + probed + + cached -> + cached + end + end + + # n_gpu_layers: 0 — the probe only needs the memory module's answer, and + # keeping it off the GPU avoids competing with the test that is about to run. + defp probe_seq_rm_kind(kind) do + :ok = LlamaCppEx.init() + {:ok, model} = LlamaCppEx.load_model(path!(kind), n_gpu_layers: 0) + {:ok, ctx} = LlamaCppEx.Context.create(model, n_ctx: 64) + LlamaCppEx.NIF.context_can_seq_rm(ctx.ref) + end + defp unavailable(kind) do {var, description} = Map.fetch!(@vars, kind) diff --git a/test/test_helper.exs b/test/test_helper.exs index 81839a8..57b21c1 100644 --- a/test/test_helper.exs +++ b/test/test_helper.exs @@ -4,6 +4,11 @@ # :smoke — generation/chat/grammar/server paths; needs LLAMA_SMOKE_GEN_MODEL # :embeddings — embedding paths; needs LLAMA_SMOKE_EMB_MODEL # :mtp — MTP speculative decoding; needs LLAMA_SMOKE_MTP_MODEL +# :mtp_cancel — one known-broken MTP test, excluded on its own tag so that +# `--include mtp` is green. It does not fail, it aborts the VM: +# cancelling an MTP stream is fire-and-forget, so reusing the +# session immediately afterwards races the still-running draft +# loop over shared contexts. See test/mtp_model_test.exs. # :slow — long-running comparison matrices (F16 vs Q8_0 KV cache); # needs LLAMA_SMOKE_GEN_MODEL # @@ -45,4 +50,4 @@ Code.require_file("support/test_models.exs", __DIR__) Code.require_file("support/test_slots.exs", __DIR__) -ExUnit.start(exclude: [:smoke, :embeddings, :slow, :mtp]) +ExUnit.start(exclude: [:smoke, :embeddings, :slow, :mtp, :mtp_cancel]) diff --git a/vendor/llama.cpp b/vendor/llama.cpp index ddd4ec1..61881b1 160000 --- a/vendor/llama.cpp +++ b/vendor/llama.cpp @@ -1 +1 @@ -Subproject commit ddd4ec1428a6201e18975ea52b07c71e0f9aef26 +Subproject commit 61881b1f7f0b13d9e46d561fc25afcd6bbaec479