diff --git a/CHANGELOG.md b/CHANGELOG.md index cbc646a..c47ab1e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,36 @@ # Changelog +## v0.8.39 + +Maintenance release: llama.cpp bump to b10133, on top of b10075 from v0.8.38. +Unlike recent bumps this range **breaks the upstream C API**, so one NIF change +was required — see the `model_load/10` entry below. The public Elixir API is +unchanged. Full suite against the rebuilt NIF with real GGUF models (smoke, slow +and MTP speculative-decoding tests all included): 252 passed, 0 failures. + +### Changed + +- **llama.cpp submodule** — Updated from 76f46ad29 to ff067f76d (58 commits, tag b10133). Two binding-relevant headers changed, one of them breaking: + - `include/llama.h` — **breaking**: `llama_model_params` loses the `use_mmap`, `use_direct_io` and `use_mlock` booleans; they are replaced by a single `enum llama_load_mode load_mode` field with values `LLAMA_LOAD_MODE_NONE` / `_MMAP` / `_MLOCK` / `_DIRECT_IO` (`none`/`mmap`/`mlock`/`dio`), plus new `llama_load_mode_name` / `llama_load_mode_from_str` helpers (#20834). Note `LLAMA_LOAD_MODE_MLOCK` means "mmap **and** mlock" — the two are no longer independent. + - `common/chat.h` — `common_chat_params::thinking_end_tag` (`std::string`) became `thinking_end_tags` (`std::vector`) so the reasoning-budget sampler can accept multiple end sequences (#25544). The binding never read that field, so no change was needed. + - `common/common.h` changes (`common_params::load_mode`, `reasoning_budget_end` widened to `std::vector`, new MCP server config fields) do not affect the binding, which does not use `common_params`. `common/json-schema-to-grammar.h`, `common/speculative.h`, `common/sampling.h` and every `ggml/include/` header are untouched in this range. + - **llama core / models**: add GLM 5.2 Indexer support (#25407); add support for Laguna XS.2 & M.1 (#25165); assorted llama bug fixes (#26051); fix DeepSeek4 APE tensor op in llama-arch (#25945); fix the crafted DeepSeek4 template (#25414); fix the reasoning-preserve variable for DS4 (#25999); cohere2 MoE template parser enforces the JSON schema for text responses when a response schema is provided (#26018); synchronize save-load-state generation in the tests (#26056). + - **common**: support multiple end sequences in the reasoning budget sampler (#25544); fix a use-after-free when loading a LoRA adapter fails (#25611); skip the empty implicit default preset (#25643); infer the speculative type from draft-repo sidecars (#25989) and resolve a draft repo to its requested sidecar (#25955). + - **ggml**: declare `gguf_writer_base`'s destructor virtual (#25867); enable PowerPC backend variants on AIX (#25983); add the `GGML_BACKEND_DL_IMPL` invocation for the OpenVINO backend (#25795). + - **Metal**: add f16 type support to leaky relu (#25981). + - **CUDA**: `GET_ROWS` for quantized types (#25962); vectorize same-type `get_rows` with an int4 copy (#25929); improve NVFP4 W4A4 activation quantization (#25730); add `sqrt_softplus` in topk-moe for dsv4 (#25896); fix external compilation of q1_0 MMQ (#25778). + - **Vulkan**: refactor `vk_queue` to use per-instance mutexes and unique handles (#23570). + - **HIP**: remove rocWMMA FlashAttention (#26046). + - **WebGPU**: add a CONV_2D_DW (depthwise conv2d) kernel (#25847); fix WASM compilation with OpenMP (#25943). + - **OpenCL**: cache compiled `cl_program` binaries on disk (#26050); do not treat NULL-mask flash attention as causal (#25771). + - **hexagon**: further pipeline improvements to the core bits (L2, DMA, MM, FA) (#26049); partial im2col support (#26007); activation ops update (#25974); check tensor type when reusing descriptors (#25968); fix a Windows crash when `op_poll` is enabled (#26029). + - **kleidiai**: warn once when a weight type has no KleidiAI kernel (#25701). + - **mtmd**: use RAII for setting and resetting non-causal attention (#25723); use `align_corners` for qwen3vl vision position embedding interpolation (#25781). + - **convert**: fix the non-MoE NomicBert GGUF conversion error (#25996); handle the HunyuanVL XD-RoPE config (#25514). + - **tools/server/ui** (not linked into the binding): MCP stdio support (#26062) and MCP display-name conflict fix (#26011); `"reasoning_effort": "none"` in the OAI API (#26045); a `format` arg on the datetime tool (#26117); missing `adaptive_target`/`adaptive_decay` task parameters in `generation_settings` (#25830); return 400 instead of 500 on validation errors with `X-Conversation-Id` (#25760); properly handle a null `llama_context` (#25868); reduced per-token render cost while streaming (#26053); assorted web UI fixes. + - **vendor / ci**: update cpp-httplib to 0.51.0 (#26067) and `subprocess.h` (#26061); fix the SYCL package shared-library lookup (#25987). +- **NIF `model_load/10`** — Now maps the existing `:use_mmap` / `:use_mlock` / `:use_direct_io` options onto the new `llama_load_mode` enum instead of setting the three removed booleans. The documented precedence is preserved (direct I/O takes precedence over mmap, and mlock implies mmap): `dio` > `mlock` > `mmap` > `none`. The Elixir API and its defaults are unchanged, so no caller updates are needed; all four resolved modes were verified against a real model load. + ## v0.8.38 Maintenance release: llama.cpp bump to b10075. Full suite against the rebuilt diff --git a/c_src/llama_cpp_ex/llama_nif.cpp b/c_src/llama_cpp_ex/llama_nif.cpp index 31ffd67..23f5971 100644 --- a/c_src/llama_cpp_ex/llama_nif.cpp +++ b/c_src/llama_cpp_ex/llama_nif.cpp @@ -143,11 +143,15 @@ model_load(ErlNifEnv* env, std::string path, int64_t n_gpu_layers, bool use_mmap bool use_mlock, bool use_direct_io, bool vocab_only, bool check_tensors) { auto params = llama_model_default_params(); params.n_gpu_layers = static_cast(n_gpu_layers); - params.use_mmap = use_mmap; params.main_gpu = static_cast(main_gpu); params.split_mode = static_cast(split_mode); - params.use_mlock = use_mlock; - params.use_direct_io = use_direct_io; + // Upstream collapsed the use_mmap/use_mlock/use_direct_io booleans into a + // single llama_load_mode enum. Preserve the documented precedence of the + // Elixir options: direct I/O wins over mmap, and mlock implies mmap. + params.load_mode = use_direct_io ? LLAMA_LOAD_MODE_DIRECT_IO + : use_mlock ? LLAMA_LOAD_MODE_MLOCK + : use_mmap ? LLAMA_LOAD_MODE_MMAP + : LLAMA_LOAD_MODE_NONE; params.vocab_only = vocab_only; params.check_tensors = check_tensors; diff --git a/docs/performance.md b/docs/performance.md index dfaa668..b09a889 100644 --- a/docs/performance.md +++ b/docs/performance.md @@ -275,12 +275,17 @@ Additional options when loading models: {:ok, model} = LlamaCppEx.load_model("model.gguf", n_gpu_layers: -1, # Offload all layers to GPU use_mmap: true, # Memory-map file (default, faster loading) - use_mlock: true, # Pin in RAM (prevent swapping) + use_mlock: true, # Pin in RAM (prevent swapping); implies use_mmap use_direct_io: false, # Bypass page cache check_tensors: true # Validate tensor data (debugging) ) ``` +These three flags resolve to llama.cpp's single `load_mode` enum, with +`use_direct_io` > `use_mlock` > `use_mmap` precedence (`dio`, `mlock`, `mmap`, +or `none` when all are false). Only one mode is ever active, so combining them +is not additive. + ## Complete Optimization Example Here's a production-ready server configuration combining multiple optimizations: diff --git a/lib/llama_cpp_ex.ex b/lib/llama_cpp_ex.ex index 6b28a5f..0f76cd8 100644 --- a/lib/llama_cpp_ex.ex +++ b/lib/llama_cpp_ex.ex @@ -42,43 +42,9 @@ defmodule LlamaCppEx do UTF8Stream } - @context_opt_keys [ - :n_threads, - :n_threads_batch, - :n_batch, - :n_ubatch, - :type_k, - :type_v, - :flash_attn, - :offload_kqv, - :op_offload, - :rope_scaling_type, - :rope_freq_base, - :rope_freq_scale, - :yarn_ext_factor, - :yarn_attn_factor, - :yarn_beta_fast, - :yarn_beta_slow, - :yarn_orig_ctx, - :attention_type, - :no_perf, - :swa_full - ] - - # Sampling options forwarded to Sampler.create/2 by the generation entry - # points. Keep in sync with the options documented on generate/3. - @sampler_opt_keys [ - :seed, - :temp, - :top_k, - :top_p, - :min_p, - :penalty_repeat, - :penalty_freq, - :penalty_present, - :grammar, - :grammar_root - ] + # Context and sampling options are owned by the modules that consume them — + # see Context.tuning_option_keys/0 and Sampler.option_keys/0. Do not copy the + # lists here; three copies had already drifted apart. # Chat-templating options split off before the rest flows to generation. @chat_opt_keys [:add_assistant, :enable_thinking, :chat_template_kwargs] @@ -266,8 +232,8 @@ defmodule LlamaCppEx do max_tokens: Keyword.get(opts, :max_tokens, 256), n_ctx: Keyword.get(opts, :n_ctx, 2048), timeout: Keyword.get(opts, :timeout, 60_000), - sampler_opts: Keyword.take(opts, @sampler_opt_keys), - ctx_opts: Keyword.take(opts, @context_opt_keys) + sampler_opts: Keyword.take(opts, Sampler.option_keys()), + ctx_opts: Keyword.take(opts, Context.tuning_option_keys()) } end diff --git a/lib/llama_cpp_ex/context.ex b/lib/llama_cpp_ex/context.ex index 2a66b98..97aa7c4 100644 --- a/lib/llama_cpp_ex/context.ex +++ b/lib/llama_cpp_ex/context.ex @@ -1,6 +1,27 @@ defmodule LlamaCppEx.Context do @moduledoc """ Inference context with KV cache. + + ## Option ownership + + This module is the single source of truth for the options `create/2` accepts. + Callers that forward user options into a context (`LlamaCppEx`, + `LlamaCppEx.Server`, `LlamaCppEx.MTP`) must select them with + `tuning_option_keys/0` rather than keeping their own copy of the list — three + hand-maintained copies had already drifted, silently dropping `:n_threads`, + `:n_threads_batch` and `:n_ubatch` on `LlamaCppEx.Server`. + + The keys are split by kind: + + * `tuning_option_keys/0` — performance knobs that are safe to forward from + any caller. They never change what the context *is*. + * `structural_option_keys/0` — options that decide the context's purpose or + size (`:embeddings`, `:pooling_type`, `:ctx_type`, `:n_ctx`, ...). Each + caller sets these explicitly; forwarding them blindly would let, say, + `embeddings: true` turn a generation server into an embedding context. + + Callers pass their own values as `[n_ctx: computed] ++ forwarded_opts`, which + wins because `Keyword.get/3` returns the first match. """ @enforce_keys [:ref, :model] @@ -8,6 +29,55 @@ defmodule LlamaCppEx.Context do @type t :: %__MODULE__{ref: reference(), model: LlamaCppEx.Model.t()} + @tuning_option_keys [ + :n_threads, + :n_threads_batch, + :n_batch, + :n_ubatch, + :type_k, + :type_v, + :flash_attn, + :offload_kqv, + :op_offload, + :rope_scaling_type, + :rope_freq_base, + :rope_freq_scale, + :yarn_ext_factor, + :yarn_attn_factor, + :yarn_beta_fast, + :yarn_beta_slow, + :yarn_orig_ctx, + :attention_type, + :no_perf, + :swa_full + ] + + @structural_option_keys [ + :n_ctx, + :n_seq_max, + :kv_unified, + :embeddings, + :pooling_type, + :ctx_type, + :n_rs_seq + ] + + @doc """ + Options that are safe for a caller to forward from user-supplied opts. + + See the "Option ownership" section in the module doc. + """ + @spec tuning_option_keys() :: [atom()] + def tuning_option_keys, do: @tuning_option_keys + + @doc """ + Options a caller must set explicitly rather than forward blindly. + + See the "Option ownership" section in the module doc. + """ + @spec structural_option_keys() :: [atom()] + def structural_option_keys, do: @structural_option_keys + @doc """ Creates a new inference context for the given model. diff --git a/lib/llama_cpp_ex/model.ex b/lib/llama_cpp_ex/model.ex index e34bbee..da9a79c 100644 --- a/lib/llama_cpp_ex/model.ex +++ b/lib/llama_cpp_ex/model.ex @@ -8,6 +8,36 @@ defmodule LlamaCppEx.Model do @type t :: %__MODULE__{ref: reference()} + @tuning_option_keys [ + :main_gpu, + :split_mode, + :tensor_split, + :use_mmap, + :use_mlock, + :use_direct_io, + :check_tensors + ] + + @structural_option_keys [:n_gpu_layers, :vocab_only] + + @doc """ + Options that are safe for a caller to forward from user-supplied opts. + + `LlamaCppEx.Server` selects its model options with this function rather than + keeping its own copy of the list. + """ + @spec tuning_option_keys() :: [atom()] + def tuning_option_keys, do: @tuning_option_keys + + @doc """ + Options a caller must set explicitly rather than forward blindly. + + `:vocab_only` in particular must never be forwarded into a server — it would + load a model with no weights. + """ + @spec structural_option_keys() :: [atom()] + def structural_option_keys, do: @structural_option_keys + @doc """ Loads a GGUF model from the given file path. @@ -21,12 +51,22 @@ defmodule LlamaCppEx.Model do Defaults to `:none`. * `:tensor_split` - List of floats specifying the proportion of work per GPU (e.g. `[0.5, 0.5]` for two GPUs). Defaults to `[]`. - * `:use_mlock` - Pin model memory in RAM to prevent swapping. Defaults to `false`. + * `:use_mlock` - Pin model memory in RAM to prevent swapping. Implies `:use_mmap`. + Defaults to `false`. * `:use_direct_io` - Bypass page cache when loading (takes precedence over mmap). Defaults to `false`. * `:vocab_only` - Load vocabulary and metadata only, skip weights. Defaults to `false`. * `:check_tensors` - Validate model tensor data on load. Defaults to `false`. + > #### Load mode {: .info} + > + > llama.cpp collapsed its three loading booleans into one `load_mode` enum, so + > these options resolve to a single mode with `:use_direct_io` > `:use_mlock` > + > `:use_mmap` precedence — respectively `dio`, `mlock`, `mmap`, and `none` when + > all are false. Because `mlock` now implies mmap upstream, passing + > `use_mlock: true, use_mmap: false` memory-maps the file rather than reading + > it into anonymous memory. + ## Examples {:ok, model} = LlamaCppEx.Model.load("path/to/model.gguf", n_gpu_layers: -1) diff --git a/lib/llama_cpp_ex/mtp.ex b/lib/llama_cpp_ex/mtp.ex index ce807ff..58bc9ad 100644 --- a/lib/llama_cpp_ex/mtp.ex +++ b/lib/llama_cpp_ex/mtp.ex @@ -52,29 +52,13 @@ defmodule LlamaCppEx.MTP do n_draft: pos_integer() } - @context_opt_keys [ - :n_threads, - :n_threads_batch, - :n_batch, - :n_ubatch, - :type_k, - :type_v, - :flash_attn, - :offload_kqv, - :op_offload, - :rope_scaling_type, - :rope_freq_base, - :rope_freq_scale, - :yarn_ext_factor, - :yarn_attn_factor, - :yarn_beta_fast, - :yarn_beta_slow, - :yarn_orig_ctx, - :attention_type, - :no_perf, - :swa_full, - :n_ctx - ] + # Context options forwarded to both the target and draft contexts. The list is + # owned by Context (tuning_option_keys/0); MTP additionally lets the caller set + # :n_ctx, which is structural everywhere else because each caller normally + # computes it. + defp forwardable_context_opts(opts) do + Keyword.take(opts, [:n_ctx | Context.tuning_option_keys()]) + end @doc """ Initializes an MTP speculative session: builds the target context, the MTP @@ -98,7 +82,7 @@ defmodule LlamaCppEx.MTP do n_draft = Keyword.get(opts, :n_draft, 3) if is_integer(n_draft) and n_draft > 0 do - base_ctx_opts = Keyword.take(opts, @context_opt_keys) + base_ctx_opts = forwardable_context_opts(opts) main_opts = Keyword.merge(base_ctx_opts, ctx_type: :default) # Match upstream server: MTP draft context is created with n_rs_seq=0. # The MTP impl handles state rollback internally via cached hidden @@ -166,19 +150,7 @@ defmodule LlamaCppEx.MTP do emit_stats_every = Keyword.get(opts, :emit_stats_every, 0) timeout = Keyword.get(opts, :timeout, 60_000) - sampler_opts = - Keyword.take(opts, [ - :seed, - :temp, - :top_k, - :top_p, - :min_p, - :penalty_repeat, - :penalty_freq, - :penalty_present, - :grammar, - :grammar_root - ]) + sampler_opts = Keyword.take(opts, Sampler.option_keys()) Stream.resource( fn -> diff --git a/lib/llama_cpp_ex/sampler.ex b/lib/llama_cpp_ex/sampler.ex index c480b49..54be73e 100644 --- a/lib/llama_cpp_ex/sampler.ex +++ b/lib/llama_cpp_ex/sampler.ex @@ -11,6 +11,30 @@ defmodule LlamaCppEx.Sampler do @type t :: %__MODULE__{ref: reference()} + @option_keys [ + :seed, + :temp, + :top_k, + :top_p, + :min_p, + :penalty_repeat, + :penalty_freq, + :penalty_present, + :grammar, + :grammar_root + ] + + @doc """ + The options `create/2` accepts. + + This module is the single source of truth: callers that forward user sampling + options (`LlamaCppEx`, `LlamaCppEx.Server`) select them with this function + instead of keeping their own copy of the list. Every sampling option is safe + to forward, so there is no tuning/structural split here. + """ + @spec option_keys() :: [atom()] + def option_keys, do: @option_keys + @doc """ Creates a new sampler chain. diff --git a/lib/llama_cpp_ex/server.ex b/lib/llama_cpp_ex/server.ex index 16194f9..cae12a4 100644 --- a/lib/llama_cpp_ex/server.ex +++ b/lib/llama_cpp_ex/server.ex @@ -155,19 +155,9 @@ defmodule LlamaCppEx.Server do alias LlamaCppEx.{Context, Model, Sampler, Tokenizer, UTF8Stream} # Sampling options accepted both at server start (defaults) and per request - # (overrides). Keep in sync with LlamaCppEx.Sampler.create/2. - @sampler_opt_keys [ - :seed, - :temp, - :top_k, - :top_p, - :min_p, - :penalty_repeat, - :penalty_freq, - :penalty_present, - :grammar, - :grammar_root - ] + # (overrides). Owned by Sampler, so this cannot drift out of sync; the + # compile-time dependency recompiles this module when that list changes. + @sampler_opt_keys Sampler.option_keys() # Per-request options accepted by generate/stream and carried through the # queue into the slot. Server-level options provide the defaults. @@ -258,6 +248,18 @@ defmodule LlamaCppEx.Server do `LlamaCppEx.Server.Strategy.DecodeMaximal`. See `LlamaCppEx.Server.BatchStrategy`. * Sampling options: `:temp`, `:top_k`, `:top_p`, `:min_p`, `:seed`, `:penalty_repeat`, `:penalty_freq`, `:penalty_present`, `:grammar`, `:grammar_root`. + * Context tuning options are forwarded to `LlamaCppEx.Context.create/2` — + `:n_threads`, `:n_threads_batch`, `:n_ubatch`, `:type_k`, `:type_v`, + `:flash_attn`, `:offload_kqv`, `:op_offload`, the RoPE/YaRN options, + `:attention_type`, `:no_perf` and `:swa_full`. See + `LlamaCppEx.Context.tuning_option_keys/0` for the authoritative list. + `:n_ctx`, `:n_batch`, `:n_seq_max` and `:kv_unified` are set by the server + from the options above and cannot be overridden here. + * Model loading options are forwarded to `LlamaCppEx.Model.load/2` — + `:main_gpu`, `:split_mode`, `:tensor_split`, `:use_mmap`, `:use_mlock`, + `:use_direct_io` and `:check_tensors`. The three load flags collapse into + llama.cpp's single `load_mode` with `:use_direct_io` > `:use_mlock` > + `:use_mmap` precedence; see `LlamaCppEx.Model.load/2`. * GenServer options like `:name`. """ @@ -509,35 +511,12 @@ defmodule LlamaCppEx.Server do sampler_opts = Keyword.take(opts, @sampler_opt_keys) - model_opts = - Keyword.take(opts, [ - :main_gpu, - :split_mode, - :tensor_split, - :use_mlock, - :use_direct_io, - :check_tensors - ]) - - context_opts = - Keyword.take(opts, [ - :type_k, - :type_v, - :flash_attn, - :offload_kqv, - :op_offload, - :rope_scaling_type, - :rope_freq_base, - :rope_freq_scale, - :yarn_ext_factor, - :yarn_attn_factor, - :yarn_beta_fast, - :yarn_beta_slow, - :yarn_orig_ctx, - :attention_type, - :no_perf, - :swa_full - ]) + # Both lists are owned by the modules that consume them. The values this + # function computes below (n_gpu_layers, n_ctx, n_batch, n_seq_max, + # kv_unified) are prepended at the call sites and win, because Keyword.get/3 + # returns the first match. + model_opts = Keyword.take(opts, Model.tuning_option_keys()) + context_opts = Keyword.take(opts, Context.tuning_option_keys()) # Trap exits so terminate/2 reliably erases the persistent_term model # cache on shutdown. diff --git a/mix.exs b/mix.exs index ca48c1c..7f0f321 100644 --- a/mix.exs +++ b/mix.exs @@ -37,7 +37,7 @@ end defmodule LlamaCppEx.MixProject do use Mix.Project - @version "0.8.38" + @version "0.8.39" @source_url "https://github.com/nyo16/llama_cpp_ex" def project do diff --git a/test/option_forwarding_test.exs b/test/option_forwarding_test.exs new file mode 100644 index 0000000..183251d --- /dev/null +++ b/test/option_forwarding_test.exs @@ -0,0 +1,173 @@ +defmodule LlamaCppEx.OptionForwardingTest do + @moduledoc """ + Drift alarm for the option key lists. + + `Context.create/2`, `Sampler.create/2` and `Model.load/2` each read a set of + options, and several callers forward user options into them. Those callers used + to keep their own copies of the key lists, and the copies drifted: the one in + `LlamaCppEx.Server` was missing `:n_threads`, `:n_threads_batch` and + `:n_ubatch`, so those options were silently dropped rather than rejected. + + These tests fail if a new option is added to one of those functions without + being classified as tuning or structural, and if any caller re-inlines its own + copy of a list. They read source rather than calling the NIF, so they need no + model and stay in the default suite. + """ + use ExUnit.Case, async: true + + alias LlamaCppEx.{Context, Model, Sampler} + + # Extracts the body of the option-reading function so sibling functions don't + # pollute the scan — Context.generate/4 also reads :max_tokens and :cancel, + # which are not Context.create/2 options. + defp function_body(path, marker) do + [_, rest] = String.split(File.read!(path), marker, parts: 2) + + ~r/\n (@doc|defp )/ + |> Regex.split(rest, parts: 2) + |> hd() + end + + # Returns the option names as strings, not atoms: String.to_existing_atom/1 + # would fail here when a setup block runs before the owning module has been + # loaded, since the atom would not exist yet. + defp opts_read_in(path, marker) do + path + |> function_body(marker) + |> then(&Regex.scan(~r/Keyword\.get(?:_lazy)?\(opts, :([a-z_0-9]+)/, &1)) + |> Enum.map(fn [_, key] -> key end) + |> Enum.uniq() + |> Enum.sort() + end + + defp declared(keys), do: keys |> Enum.map(&Atom.to_string/1) |> Enum.sort() + + describe "Context.create/2" do + setup do + %{read: opts_read_in("lib/llama_cpp_ex/context.ex", "def create(")} + end + + test "every option it reads is classified as tuning or structural", %{read: read} do + declared = declared(Context.tuning_option_keys() ++ Context.structural_option_keys()) + + assert read -- declared == [], + """ + Context.create/2 reads options that are in neither + tuning_option_keys/0 nor structural_option_keys/0: #{inspect(read -- declared)} + + Classify each new option. Tuning keys are forwarded by callers + automatically; structural keys must be set explicitly by each caller. + """ + end + + test "every declared key is actually read by create/2", %{read: read} do + declared = declared(Context.tuning_option_keys() ++ Context.structural_option_keys()) + + assert declared -- read == [], + "declared but never read by Context.create/2: #{inspect(declared -- read)}" + end + + test "tuning and structural sets are disjoint" do + # A -- (A -- B) is the intersection. + overlap = + Context.tuning_option_keys() -- + (Context.tuning_option_keys() -- Context.structural_option_keys()) + + assert overlap == [], + "keys classified as both tuning and structural: #{inspect(overlap)}" + end + + test "context-defining options are never forwardable" do + for key <- [:embeddings, :pooling_type, :ctx_type, :n_ctx] do + refute key in Context.tuning_option_keys(), + """ + #{inspect(key)} must stay structural. Forwarding it would let a caller + change what the context is — e.g. embeddings: true would turn a + generation server into an embedding context. + """ + end + end + end + + describe "Sampler.create/2" do + test "option_keys/0 matches exactly the options it reads" do + read = opts_read_in("lib/llama_cpp_ex/sampler.ex", "def create(") + + assert read == declared(Sampler.option_keys()) + end + end + + describe "Model.load/2" do + setup do + %{read: opts_read_in("lib/llama_cpp_ex/model.ex", "def load(")} + end + + test "every option it reads is classified as tuning or structural", %{read: read} do + declared = declared(Model.tuning_option_keys() ++ Model.structural_option_keys()) + + assert read -- declared == [], + "Model.load/2 reads unclassified options: #{inspect(read -- declared)}" + end + + test "every declared key is actually read by load/2", %{read: read} do + declared = declared(Model.tuning_option_keys() ++ Model.structural_option_keys()) + + assert declared -- read == [], + "declared but never read by Model.load/2: #{inspect(declared -- read)}" + end + + test "vocab_only is never forwardable" do + refute :vocab_only in Model.tuning_option_keys(), + ":vocab_only must stay structural — forwarding it into a server would " <> + "load a model with no weights." + end + + test "all three load-mode flags are forwardable together" do + # They collapse into llama.cpp's single load_mode enum with + # dio > mlock > mmap > none precedence. Forwarding a subset (Server used to + # omit :use_mmap) leaves callers unable to select some modes. + for key <- [:use_mmap, :use_mlock, :use_direct_io] do + assert key in Model.tuning_option_keys() + end + end + end + + describe "callers do not keep their own copies" do + @callers [ + "lib/llama_cpp_ex.ex", + "lib/llama_cpp_ex/server.ex", + "lib/llama_cpp_ex/mtp.ex" + ] + + # Matches a literal multi-atom list passed to Keyword.take/2 — `[:a, :b]` — + # which is how every copy drifted. Deliberately does not match the cons form + # `[:n_ctx | Context.tuning_option_keys()]` (MTP's one extra key), nor the + # option names that legitimately appear in @doc prose. + @inline_list ~r/Keyword\.take\([a-z_]+,\s*\[\s*:[a-z_0-9]+\s*,/ + + test "no caller passes an inline literal key list to Keyword.take/2" do + for path <- @callers do + refute Regex.match?(@inline_list, File.read!(path)), + """ + #{path} passes a literal key list to Keyword.take/2. That is how the + copies drifted before — select the keys from the module that owns + them: Context.tuning_option_keys/0, Sampler.option_keys/0 or + Model.tuning_option_keys/0. + """ + end + end + + test "Server selects its option sets from the owning modules" do + source = File.read!("lib/llama_cpp_ex/server.ex") + + for expected <- [ + "Context.tuning_option_keys()", + "Model.tuning_option_keys()", + "Sampler.option_keys()" + ] do + assert String.contains?(source, expected), + "server.ex should select options via #{expected}" + end + end + end +end diff --git a/test/server_smoke_test.exs b/test/server_smoke_test.exs index 496fe99..f5f5021 100644 --- a/test/server_smoke_test.exs +++ b/test/server_smoke_test.exs @@ -87,8 +87,13 @@ defmodule LlamaCppEx.ServerSmokeTest do server = start_server([]) attach_collector([:llama_cpp_ex, :server, :request, :start]) - a1 = "Chat A. User: Name three colors.\nAssistant:" - b1 = "Chat B, unrelated. User: Name three animals.\nAssistant:" + # The two prompts must share no leading tokens. pick_cached_slot/2 reuses a + # slot when longest-common-prefix / prompt_len > 0.1, so a shared opener + # (both prompts used to start with "Chat ") is enough to route b1 onto a1's + # slot on some tokenizers — this test then failed on Llama-3.2 while + # passing on Qwen3.5. Keep the first tokens distinct. + a1 = "Weather log. User: Name three colors.\nAssistant:" + b1 = "Zoology notes, unrelated. User: Name three animals.\nAssistant:" {:ok, ra} = Server.generate(server, a1, max_tokens: 12, session: :a) {_, %{seq_id: slot_a}} = next_telemetry() diff --git a/vendor/llama.cpp b/vendor/llama.cpp index 76f46ad..ff067f7 160000 --- a/vendor/llama.cpp +++ b/vendor/llama.cpp @@ -1 +1 @@ -Subproject commit 76f46ad29d61fd8c1401e8221842934bf62a6064 +Subproject commit ff067f76dd8e9e05f0528056f1274adf01a54d70