Review remediation: close all 3 blockers and 27 warnings/suggestions - #79
Merged
Conversation
Implements .claude/plans/audit-remediation/plan.md across 9 phases: CI
verification gate, VM safety at the NIF boundary, public-API error
contracts, release pipeline, HuggingFace Hub hardening, OTP hygiene, test
integrity, performance, and two architecture extractions.
Breaking (see CHANGELOG for the full list):
- chat_completion/3 returns {:error, reason} where it previously reported a
failed generation as {:ok, %ChatCompletion{finish_reason: "stop"}}
- the four facade streams emit a final {:error, reason} element instead of
halting silently, matching Server.stream/3
- unknown options raise ArgumentError at every public entry point
- Server.get_model/1 raises instead of exiting; new fetch_model/1
- Server :max_queue defaults to 64, not 0
- ModelManager.list/0 and default/0 return {:error, :not_started}
- %Sampler{} gained :model; the library now starts a supervision tree
Gates: compile --warnings-as-errors clean, format clean, credo --strict 0
issues, dialyzer 0 errors, 320 passed / 118 excluded, 417 passed with
--include smoke --include embeddings (8 seeds), and the Hex-tarball source
build that was impossible before now exits 0.
KNOWN BLOCKERS — do not read this commit as a clean bill of health. A
subsequent /phx:review found three, planned in
.claude/plans/audit-review-fixes/plan.md:
1. Three llama_memory_seq_rm call sites still take an unchecked seq_id from
Elixir and reach ggml_abort(): embed_decode/4, embed_batch_decode/2, and
batch_eval_sample/4's purgeable_seq_ids. The CHANGELOG's NIF section has
been corrected to name them rather than claim the boundary is closed.
2. MTP.stream_events/3 returns an infinite stream on setup failure; a bad
grammar makes MTP.generate/3 hang.
3. init_slot/4 still hard-matches Sampler.create/2, which this change set
made fallible, so a caller-supplied malformed grammar crashes the
model-holding GenServer (restart: :temporary, so it does not come back).
Blockers 2 and 3 are the same defect as each other and as two of the
warnings: a return type was widened and a consumer was missed. The
follow-up plan adds the mechanical guards for that class rather than only
patching the call sites, because manual review missed it four times.
Phases 1-3 of .claude/plans/audit-review-fixes/plan.md, addressing the three
BLOCKERs and 11 WARNINGs from the audit-remediation review.
Widened contracts (B-3, W-9, W-12, W-13)
New validate-only `grammar_validate` NIF plus `Sampler.validate_grammar/2`, so
a caller-supplied `:grammar` is rejected at the admission boundary instead of
crashing the GenServer that owns the model. `init_slot/4` splits into a
sampler-first `init_slot/4` and `install_slot/5`, with `reject_request/3` as
depth-2 defence that fails the request rather than the server.
`@server_opt_keys` is built from what the server accepts, not from
`@gen_opt_keys` — 21 keys used to pass the outer gate and raise inside
`Server.complete_tokens/3`, naming a function the user never called.
`subscribe_stream_tokens/4` now validates its options and its `@spec` admits
the `{:error, _}` its only caller depends on. `ModelIO` routes with two
allowlists plus an outer gate on their union, so `:vocab_only` stops reaching
`Server.start_link/1` without letting `n_paralell` through silently.
A containment test in option_forwarding_test.exs asserts the invariant all
four instances violated. It found a fifth on its first run: `:template` was
documented on chat/3 and chat_completion/3 and nothing ever read it.
NIF boundary (B-1, W-7, W-1, W-20)
The three `llama_memory_seq_rm` call sites that took an unchecked seq_id
outside any batch are guarded. All seven `llama_decode` calls that bypassed
`decode_tracked` now go through it, so the bound `sampler_sample_at` enforces
can no longer go stale in the permissive direction. `memory_seq_rm` returning
false is handled instead of hard-matched. nif_guards_test.exs replaces its
hand-picked subset with a `@seq_id_surface` table checked against nif.ex.
Streaming (B-2, W-5, W-6, W-8, W-10, W-11)
`MTP.stream_events/3` no longer returns an infinite stream on setup failure.
A timed-out server-routed stream cancels its request instead of leaking the
slot. `Server.stream/3` and `stream_tokens/3` emit `{:error, :timeout}`
instead of truncating. `Generator.stop/1` drains the runner's exit signal.
`fetch_model/1` and the streaming admission calls turn every exit into a
value, and are bounded by the caller's `:timeout` rather than 5000 ms.
…duals
Phases 4-5 of .claude/plans/audit-review-fixes/plan.md.
Swallowed errors (W-4, W-18, W-14)
`PromptCache.save/4` splits the NIF-failure clause out of the four policy
declines and logs it, so "the RAM cache never populates" is diagnosable.
Device enumeration only degrades on `:not_loaded`; any other `ErlangError`
propagates instead of being reported as "this machine has no GPUs".
The download write path had a different bug than the review described:
`IO.binwrite/2` does not return `{:error, reason}`, it calls
`:erlang.error(reason)`, so a full disk raised out of Req's streaming callback
and past `download/3` into the caller's crash report — request options, `:token`
included, in the stacktrace. It now goes through `:file.write/2`. Req's retry is
off for this request: it re-runs with the same `into:` closure over the
still-open device, appending a second response body rather than restarting the
file.
Hub and affinity residuals (W-16, W-17, W-19)
Integrity verification no longer downgrades itself when the metadata response
omits `lfs.sha256`; `:verify_checksum` is a deliberate three-state policy
(`true` | `:best_effort` | `false`) and anything else is rejected rather than
treated as truthy. `revision` joins the cache key, so a pin pins. Session
affinity keys on `{cache_scope, session}`, so a guessed session id can no longer
claim another scope's slot — verified both ways against a live server.
Phase 6 of .claude/plans/audit-review-fixes/plan.md.
W-21 The two embedding tests carried two gate tags (`:smoke` from the module,
`:embeddings` per test). `--include` beats `--exclude`, so `--include smoke`
alone pulled them in and failed for want of LLAMA_SMOKE_EMB_MODEL; CI masked
it by happening to pass both flags. They move to test/embeddings_test.exs
with one gate tag, and stop paying for SmokeTest's generation model.
W-22 `:slow` was excluded by default and included by nothing, so 10 KV
quantization cases had never executed anywhere. CI now includes it — the
model it needs is already cached — and test_helper.exs names that model.
Running them for the first time surfaced a pre-existing bad assertion:
`byte_size(text) > 0` measures the model's willingness to speak, not the KV
cache, and greedy decoding of "What is 2 + 2?" on SmolLM2-135M legitimately
emits EOG first. It asserts `is_binary/1` and valid UTF-8, which is what the
suite's own moduledoc says it defends.
CI also now runs each gate tag on its own, so the one-tag-per-suite
invariant is checked rather than trusted.
W-15 The TestSlots drift test `base_fields/0` was exported for, and which had
zero callers. Break-verified against a field added to `idle_slot_fields/3`.
W-23 "the child is temporary" asserted `active == 0` after a failed start, which
holds for any `restart:` value. A stub standing in for the DynamicSupervisor
captures the real child spec instead, and also pins that no hub or manager
option reaches `Server.start_link/1`.
Coverage `Server.Request` (whole module), `Options` timeouts including the
`:stream` clause that likely never ran, `ModelManager.started?/0`,
`PromptCache.restore/4` including the branch that prevents garbage KV, and
`Generator` — where W-8 lived, which was not a coincidence.
Phase 7 of .claude/plans/audit-review-fixes/plan.md, plus a determinism fix to the
W-5/W-6 tests added in Phase 3.
S-1 The NIF validation header claimed every value crossing the boundary is
range-checked *there*. Half are: values reaching a `llama_memory_*` or
`llama_state_seq_*` call directly. The rest are bounded by upstream's
`llama_batch_allocr::init`, which returns false rather than asserting — a
load-bearing check that was invisible from this file, and part of why the
three unguarded sites looked closed.
S-2 The discard-vs-`true =` split on `memory_seq_rm` now has a stated rule: a
full clear always succeeds so its result carries no information; a partial
trim can be refused and must be checked.
S-3 `init/1` claimed `trap_exit` keeps a linked process's death from taking the
model down; the `{:EXIT, _, reason}` clause does exactly the opposite. The
two readings imply opposite failure modes for anything that links itself to
a server. Corrected, and the landmine named.
S-4 The `case` glob in the semver guard is load-bearing, not belt-and-braces:
`grep -Eq` is line-oriented and would pass a two-line value on its own.
S-5 Both files already carried the OTP wording; what a reader could not see was
the Elixir floor, which lived only in mix.exs.
S-6 Covered by W-14's comment in `stream_into_device/3`.
Test determinism: the three W-5/W-6 timeout tests forced their timeout by racing
a decode with `timeout: 1`, which a consumer that keeps up simply wins — it finds
each token already in its mailbox and the stream completes with no timeout. They
now fill the only slot first, so the stream under test waits in the queue with
nothing to read. W-5's assertion also moved earlier: it reads `queue_depth`
before the hog finishes, which is exact because `cancel/2` is a cast and
`get_stats/1` a subsequent call from the same process. Break-verified: reverting
W-5 fails exactly one test, reverting W-6 fails three.
Lost from the Phase 7 commit: a break-verification for W-6 ran `git checkout -- lib/llama_cpp_ex/server.ex` while this edit was still uncommitted, so 6564f5c shipped every other S-* change but not this one. Comments only, which is why no gate caught it. `init/1` claimed trap_exit is on "so that a linked stream consumer's death does not take the model down with it". The `{:EXIT, _, reason}` clause does the exact opposite: it honours the signal with `{:stop, reason, state}`, so a link that breaks takes the model down. The two readings imply opposite failure modes for anything that links itself to a server, which is why leaving both in place was worse than either one alone. Corrected to the real reason — terminate/2 on shutdown — and the handle_info clause now names the landmine: every exit signal stops the server, `:normal` included, and `Generator` used to leave exactly such a signal behind because an already-queued `{:EXIT, pid, :normal}` survives `Process.unlink/1`. A server driving a generator would have shut down with reason `:normal` and no log line.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Remediates
/phx:review's BLOCKED verdict on the v0.8.40 audit-remediationchange set. Executes
.claude/plans/audit-review-fixes/plan.mdin full: 33 of 33tasks, covering 3 blockers, 21 warnings and 6 suggestions. The 12 deferred
suggestions are untouched and still need your call.
v0.8.40 is unreleased (latest tag is v0.8.25), so this folds into that section of
the CHANGELOG rather than inventing a version.
The three blockers
{:ok, sampler} = Sampler.create(...)insideinit_slot/4crashed a model-holding GenServer on a caller-supplied:grammar. Backing servers runrestart: :temporary, sogrammar_validateNIF (no chain allocated) +Sampler.validate_grammar/2. All five public entry points andsubscribe_stream_tokens/4reject atinit_slot/4keeps acaseas depth-2 defence that fails the request.llama_memory_seq_rmcall sites took an uncheckedseq_idoutside any batch, so upstream's validation never saw it andGGML_ASSERTaborted the VM.embed_decode's guard exits the OS process withSIGABRT.MTP.stream_events/3returned an infinite stream on setup failure — failure was signalled by adding a:setup_errorkey instead of flipping the discriminant, so thegenerate/3'sEnum.to_list/1hung with a growing heap.Taskso a recurrence failsTwo guards, because manual review missed these classes repeatedly
The widened-contract class was missed four times, three of them by the change
set that was fixing it. The NIF gap survived a break-verification that only proved
the guards someone had already thought of were load-bearing.
option_forwarding_test.exsnow asserts the containment invariant itself:every key an outer gate accepts is accepted by every inner gate it routes to,
except what the outer consumes. Break-verified in both directions.
nif_guards_test.exsreplaces a hand-picked subset (7 of 15seq_id-bearingNIFs — missing exactly the three unguarded ones) with a
@seq_id_surfacetableof 17 argument positions and four outcome classes, checked against
nif.ex's ownsource so a new
seq_id-taking NIF fails the suite until it is classified.The containment guard found a fifth instance on its first run:
:templatewasdocumented on
chat/3andchat_completion/3, accepted by@chat_opt_keys, andread by nothing. Removed.
Three findings the remediation itself turned up
IO.binwrite/2does not return{:error, reason}— it calls:erlang.error(reason). SoW-14's stated mechanism was wrong and the blastradius was larger: a full disk raised out of Req's streaming callback and past
download/3into the caller's crash report, with:tokenin the stacktrace.Switched to
:file.write/2.kv_quantization_test.exsassertedbyte_size(text) > 0, which measures themodel's willingness to speak rather than the KV cache. Pre-existing and invisible
because
:slowhad never run anywhere — and a blocker for enabling it in CI.ModelManager.load/3, because the denylistW-13replaced was doubling as thetypo gate. Closed with an outer gate on the union.
Breaking changes
All in the unreleased v0.8.40 section, each with its rationale:
<repo_id>/<revision>/<filename>—revisionwas absent fromthe key, so
revision: "<sha>"returned the cachedmaincopy and the pin boughtnothing. Existing caches re-download once.
:verify_checksumis
true(fail closed) /:best_effort/false; anything else is rejectedrather than treated as truthy. Verification used to be downgradable by the
metadata response — strip one JSON key and the bytes were cached unverified.
Server.stream/3andstream_tokens/3emit{:error, :timeout}instead oftruncating silently, and cancel server-side.
:templateis gone (see above).from inside
Server.complete_tokens/3— a function the caller never called.Verification
Every gate green, and every new acceptance criterion break-verified.
mix compile --warnings-as-errors --forcemix format --check-formattedmix credo --strictmix dialyzermix test--include smoke --include embeddings --include slow:slownever ran--include smoke, GEN model only--include slow, GEN model only--include embeddings, EMB model onlydevices/0works51 tests added to the default suite (320 → 371), 82 across the full matrix
(417 → 499). No
erl_crash.dumpfrom any test run.Break-verifications performed: containment test (both directions + orphan keys),
enumerated NIF surface (
SIGABRTon guard removal, coverage test on a new NIF),TestSlotsdrift, B-2 clause order, W-5 cancel-on-abandon, W-6 timeout element.CI additionally now runs each gate tag alone, so the one-tag-per-suite
invariant the taxonomy rests on is checked rather than trusted.
Two things to know
W-7reroutes threellama_decodecalls in theMTP path through
decode_tracked, and no MTP-capable GGUF is available locally(
Qwen3.5-0.8Bhas no MTP head —mtp_test.exsfails inContext.createbeforeany decode, both before and after this change). The three sites are a provable
no-op on decode semantics:
sp.ctx_tgt->ctxis the same pointer the local held.W-4's error branch is not directly exercised. It is reachable only from astate_seq_get_datafailure, andsave/4checksstate_seq_get_size > 0first,so the two would have to disagree. There is no NIF-level fault injector in the
suite.