Skip to content

Review remediation: close all 3 blockers and 27 warnings/suggestions - #79

Merged
nyo16 merged 6 commits into
masterfrom
review-remediation
Jul 27, 2026
Merged

Review remediation: close all 3 blockers and 27 warnings/suggestions#79
nyo16 merged 6 commits into
masterfrom
review-remediation

Conversation

@nyo16

@nyo16 nyo16 commented Jul 27, 2026

Copy link
Copy Markdown
Owner

Remediates /phx:review's BLOCKED verdict on the v0.8.40 audit-remediation
change set. Executes .claude/plans/audit-review-fixes/plan.md in full: 33 of 33
tasks
, covering 3 blockers, 21 warnings and 6 suggestions. The 12 deferred
suggestions are untouched and still need your call.

v0.8.40 is unreleased (latest tag is v0.8.25), so this folds into that section of
the CHANGELOG rather than inventing a version.

The three blockers

Defect Fix
B-3 {:ok, sampler} = Sampler.create(...) inside init_slot/4 crashed a model-holding GenServer on a caller-supplied :grammar. Backing servers run restart: :temporary, so
it never came back. New validate-only grammar_validate NIF (no chain allocated) + Sampler.validate_grammar/2. All five public entry points and subscribe_stream_tokens/4 reject at
admission; init_slot/4 keeps a case as depth-2 defence that fails the request.
B-1 Three llama_memory_seq_rm call sites took an unchecked seq_id outside any batch, so upstream's validation never saw it and GGML_ASSERT aborted the VM. Guarded,
each keeping its NIF's existing error type. Break-verified: removing embed_decode's guard exits the OS process with SIGABRT.
B-2 MTP.stream_events/3 returned an infinite stream on setup failure — failure was signalled by adding a :setup_error key instead of flipping the discriminant, so the
halt clause never matched and generate/3's Enum.to_list/1 hung with a growing heap. Halt clause ordered first, with regression tests driven through a Task so a recurrence fails
instead of wedging the suite.

Two guards, because manual review missed these classes repeatedly

The widened-contract class was missed four times, three of them by the change
set that was fixing it
. The NIF gap survived a break-verification that only proved
the guards someone had already thought of were load-bearing.

  • option_forwarding_test.exs now asserts the containment invariant itself:
    every key an outer gate accepts is accepted by every inner gate it routes to,
    except what the outer consumes. Break-verified in both directions.
  • nif_guards_test.exs replaces a hand-picked subset (7 of 15 seq_id-bearing
    NIFs — missing exactly the three unguarded ones) with a @seq_id_surface table
    of 17 argument positions and four outcome classes, checked against nif.ex's own
    source so a new seq_id-taking NIF fails the suite until it is classified.

The containment guard found a fifth instance on its first run: :template was
documented on chat/3 and chat_completion/3, accepted by @chat_opt_keys, and
read by nothing. Removed.

Three findings the remediation itself turned up

  1. IO.binwrite/2 does not return {:error, reason} — it calls
    :erlang.error(reason). So W-14's stated mechanism was wrong and the blast
    radius was larger: a full disk raised out of Req's streaming callback and past
    download/3 into the caller's crash report, with :token in the stacktrace.
    Switched to :file.write/2.
  2. kv_quantization_test.exs asserted byte_size(text) > 0, which measures the
    model's willingness to speak rather than the KV cache. Pre-existing and invisible
    because :slow had never run anywhere — and a blocker for enabling it in CI.
  3. Two allowlists alone would have swallowed option typos in
    ModelManager.load/3, because the denylist W-13 replaced was doubling as the
    typo gate. Closed with an outer gate on the union.

Breaking changes

All in the unreleased v0.8.40 section, each with its rationale:

  • Hub caches under <repo_id>/<revision>/<filename>revision was absent from
    the key, so revision: "<sha>" returned the cached main copy and the pin bought
    nothing. Existing caches re-download once.
  • A download whose file has no published SHA-256 now fails. :verify_checksum
    is true (fail closed) / :best_effort / false; anything else is rejected
    rather than treated as truthy. Verification used to be downgradable by the
    metadata response — strip one JSON key and the bytes were cached unverified.
  • Server.stream/3 and stream_tokens/3 emit {:error, :timeout} instead of
    truncating silently, and cancel server-side.
  • :template is gone (see above).
  • The server-routed facade rejects start-time options up front instead of raising
    from inside Server.complete_tokens/3 — a function the caller never called.

Verification

Every gate green, and every new acceptance criterion break-verified.

Gate Baseline Now
mix compile --warnings-as-errors --force clean clean
mix format --check-formatted clean clean
mix credo --strict 0 issues 0 issues (768 mods/funs)
mix dialyzer 0 errors 0 errors
mix test 320 / 118 excluded 371 / 139 excluded
--include smoke --include embeddings --include slow :slow never ran 499 / 11 excluded
--include smoke, GEN model only failed (W-21) 479 / 31 excluded
--include slow, GEN model only never run 381 / 129 excluded
--include embeddings, EMB model only n/a 381 / 129 excluded
C1 Hex-tarball source build rc=0 rc=0, NIF loads, devices/0 works

51 tests added to the default suite (320 → 371), 82 across the full matrix
(417 → 499). No erl_crash.dump from any test run.

Break-verifications performed: containment test (both directions + orphan keys),
enumerated NIF surface (SIGABRT on guard removal, coverage test on a new NIF),
TestSlots drift, B-2 clause order, W-5 cancel-on-abandon, W-6 timeout element.

CI additionally now runs each gate tag alone, so the one-tag-per-suite
invariant the taxonomy rests on is checked rather than trusted.

Two things to know

  • MTP is compile-verified only. W-7 reroutes three llama_decode calls in the
    MTP path through decode_tracked, and no MTP-capable GGUF is available locally
    (Qwen3.5-0.8B has no MTP head — mtp_test.exs fails in Context.create before
    any decode, both before and after this change). The three sites are a provable
    no-op on decode semantics: sp.ctx_tgt->ctx is the same pointer the local held.
  • W-4's error branch is not directly exercised. It is reachable only from a
    state_seq_get_data failure, and save/4 checks state_seq_get_size > 0 first,
    so the two would have to disagree. There is no NIF-level fault injector in the
    suite.

nyo16 added 6 commits July 27, 2026 12:57
Implements .claude/plans/audit-remediation/plan.md across 9 phases: CI
verification gate, VM safety at the NIF boundary, public-API error
contracts, release pipeline, HuggingFace Hub hardening, OTP hygiene, test
integrity, performance, and two architecture extractions.

Breaking (see CHANGELOG for the full list):
- chat_completion/3 returns {:error, reason} where it previously reported a
  failed generation as {:ok, %ChatCompletion{finish_reason: "stop"}}
- the four facade streams emit a final {:error, reason} element instead of
  halting silently, matching Server.stream/3
- unknown options raise ArgumentError at every public entry point
- Server.get_model/1 raises instead of exiting; new fetch_model/1
- Server :max_queue defaults to 64, not 0
- ModelManager.list/0 and default/0 return {:error, :not_started}
- %Sampler{} gained :model; the library now starts a supervision tree

Gates: compile --warnings-as-errors clean, format clean, credo --strict 0
issues, dialyzer 0 errors, 320 passed / 118 excluded, 417 passed with
--include smoke --include embeddings (8 seeds), and the Hex-tarball source
build that was impossible before now exits 0.

KNOWN BLOCKERS — do not read this commit as a clean bill of health. A
subsequent /phx:review found three, planned in
.claude/plans/audit-review-fixes/plan.md:

1. Three llama_memory_seq_rm call sites still take an unchecked seq_id from
   Elixir and reach ggml_abort(): embed_decode/4, embed_batch_decode/2, and
   batch_eval_sample/4's purgeable_seq_ids. The CHANGELOG's NIF section has
   been corrected to name them rather than claim the boundary is closed.
2. MTP.stream_events/3 returns an infinite stream on setup failure; a bad
   grammar makes MTP.generate/3 hang.
3. init_slot/4 still hard-matches Sampler.create/2, which this change set
   made fallible, so a caller-supplied malformed grammar crashes the
   model-holding GenServer (restart: :temporary, so it does not come back).

Blockers 2 and 3 are the same defect as each other and as two of the
warnings: a return type was widened and a consumer was missed. The
follow-up plan adds the mechanical guards for that class rather than only
patching the call sites, because manual review missed it four times.
Phases 1-3 of .claude/plans/audit-review-fixes/plan.md, addressing the three
BLOCKERs and 11 WARNINGs from the audit-remediation review.

Widened contracts (B-3, W-9, W-12, W-13)
  New validate-only `grammar_validate` NIF plus `Sampler.validate_grammar/2`, so
  a caller-supplied `:grammar` is rejected at the admission boundary instead of
  crashing the GenServer that owns the model. `init_slot/4` splits into a
  sampler-first `init_slot/4` and `install_slot/5`, with `reject_request/3` as
  depth-2 defence that fails the request rather than the server.

  `@server_opt_keys` is built from what the server accepts, not from
  `@gen_opt_keys` — 21 keys used to pass the outer gate and raise inside
  `Server.complete_tokens/3`, naming a function the user never called.
  `subscribe_stream_tokens/4` now validates its options and its `@spec` admits
  the `{:error, _}` its only caller depends on. `ModelIO` routes with two
  allowlists plus an outer gate on their union, so `:vocab_only` stops reaching
  `Server.start_link/1` without letting `n_paralell` through silently.

  A containment test in option_forwarding_test.exs asserts the invariant all
  four instances violated. It found a fifth on its first run: `:template` was
  documented on chat/3 and chat_completion/3 and nothing ever read it.

NIF boundary (B-1, W-7, W-1, W-20)
  The three `llama_memory_seq_rm` call sites that took an unchecked seq_id
  outside any batch are guarded. All seven `llama_decode` calls that bypassed
  `decode_tracked` now go through it, so the bound `sampler_sample_at` enforces
  can no longer go stale in the permissive direction. `memory_seq_rm` returning
  false is handled instead of hard-matched. nif_guards_test.exs replaces its
  hand-picked subset with a `@seq_id_surface` table checked against nif.ex.

Streaming (B-2, W-5, W-6, W-8, W-10, W-11)
  `MTP.stream_events/3` no longer returns an infinite stream on setup failure.
  A timed-out server-routed stream cancels its request instead of leaking the
  slot. `Server.stream/3` and `stream_tokens/3` emit `{:error, :timeout}`
  instead of truncating. `Generator.stop/1` drains the runner's exit signal.
  `fetch_model/1` and the streaming admission calls turn every exit into a
  value, and are bounded by the caller's `:timeout` rather than 5000 ms.
…duals

Phases 4-5 of .claude/plans/audit-review-fixes/plan.md.

Swallowed errors (W-4, W-18, W-14)
  `PromptCache.save/4` splits the NIF-failure clause out of the four policy
  declines and logs it, so "the RAM cache never populates" is diagnosable.
  Device enumeration only degrades on `:not_loaded`; any other `ErlangError`
  propagates instead of being reported as "this machine has no GPUs".

  The download write path had a different bug than the review described:
  `IO.binwrite/2` does not return `{:error, reason}`, it calls
  `:erlang.error(reason)`, so a full disk raised out of Req's streaming callback
  and past `download/3` into the caller's crash report — request options, `:token`
  included, in the stacktrace. It now goes through `:file.write/2`. Req's retry is
  off for this request: it re-runs with the same `into:` closure over the
  still-open device, appending a second response body rather than restarting the
  file.

Hub and affinity residuals (W-16, W-17, W-19)
  Integrity verification no longer downgrades itself when the metadata response
  omits `lfs.sha256`; `:verify_checksum` is a deliberate three-state policy
  (`true` | `:best_effort` | `false`) and anything else is rejected rather than
  treated as truthy. `revision` joins the cache key, so a pin pins. Session
  affinity keys on `{cache_scope, session}`, so a guessed session id can no longer
  claim another scope's slot — verified both ways against a live server.
Phase 6 of .claude/plans/audit-review-fixes/plan.md.

W-21  The two embedding tests carried two gate tags (`:smoke` from the module,
      `:embeddings` per test). `--include` beats `--exclude`, so `--include smoke`
      alone pulled them in and failed for want of LLAMA_SMOKE_EMB_MODEL; CI masked
      it by happening to pass both flags. They move to test/embeddings_test.exs
      with one gate tag, and stop paying for SmokeTest's generation model.

W-22  `:slow` was excluded by default and included by nothing, so 10 KV
      quantization cases had never executed anywhere. CI now includes it — the
      model it needs is already cached — and test_helper.exs names that model.
      Running them for the first time surfaced a pre-existing bad assertion:
      `byte_size(text) > 0` measures the model's willingness to speak, not the KV
      cache, and greedy decoding of "What is 2 + 2?" on SmolLM2-135M legitimately
      emits EOG first. It asserts `is_binary/1` and valid UTF-8, which is what the
      suite's own moduledoc says it defends.

      CI also now runs each gate tag on its own, so the one-tag-per-suite
      invariant is checked rather than trusted.

W-15  The TestSlots drift test `base_fields/0` was exported for, and which had
      zero callers. Break-verified against a field added to `idle_slot_fields/3`.

W-23  "the child is temporary" asserted `active == 0` after a failed start, which
      holds for any `restart:` value. A stub standing in for the DynamicSupervisor
      captures the real child spec instead, and also pins that no hub or manager
      option reaches `Server.start_link/1`.

Coverage  `Server.Request` (whole module), `Options` timeouts including the
      `:stream` clause that likely never ran, `ModelManager.started?/0`,
      `PromptCache.restore/4` including the branch that prevents garbage KV, and
      `Generator` — where W-8 lived, which was not a coincidence.
Phase 7 of .claude/plans/audit-review-fixes/plan.md, plus a determinism fix to the
W-5/W-6 tests added in Phase 3.

S-1  The NIF validation header claimed every value crossing the boundary is
     range-checked *there*. Half are: values reaching a `llama_memory_*` or
     `llama_state_seq_*` call directly. The rest are bounded by upstream's
     `llama_batch_allocr::init`, which returns false rather than asserting — a
     load-bearing check that was invisible from this file, and part of why the
     three unguarded sites looked closed.
S-2  The discard-vs-`true =` split on `memory_seq_rm` now has a stated rule: a
     full clear always succeeds so its result carries no information; a partial
     trim can be refused and must be checked.
S-3  `init/1` claimed `trap_exit` keeps a linked process's death from taking the
     model down; the `{:EXIT, _, reason}` clause does exactly the opposite. The
     two readings imply opposite failure modes for anything that links itself to
     a server. Corrected, and the landmine named.
S-4  The `case` glob in the semver guard is load-bearing, not belt-and-braces:
     `grep -Eq` is line-oriented and would pass a two-line value on its own.
S-5  Both files already carried the OTP wording; what a reader could not see was
     the Elixir floor, which lived only in mix.exs.
S-6  Covered by W-14's comment in `stream_into_device/3`.

Test determinism: the three W-5/W-6 timeout tests forced their timeout by racing
a decode with `timeout: 1`, which a consumer that keeps up simply wins — it finds
each token already in its mailbox and the stream completes with no timeout. They
now fill the only slot first, so the stream under test waits in the queue with
nothing to read. W-5's assertion also moved earlier: it reads `queue_depth`
before the hog finishes, which is exact because `cancel/2` is a cast and
`get_stats/1` a subsequent call from the same process. Break-verified: reverting
W-5 fails exactly one test, reverting W-6 fails three.
Lost from the Phase 7 commit: a break-verification for W-6 ran
`git checkout -- lib/llama_cpp_ex/server.ex` while this edit was still
uncommitted, so 6564f5c shipped every other S-* change but not this one. Comments
only, which is why no gate caught it.

`init/1` claimed trap_exit is on "so that a linked stream consumer's death does
not take the model down with it". The `{:EXIT, _, reason}` clause does the exact
opposite: it honours the signal with `{:stop, reason, state}`, so a link that
breaks takes the model down. The two readings imply opposite failure modes for
anything that links itself to a server, which is why leaving both in place was
worse than either one alone.

Corrected to the real reason — terminate/2 on shutdown — and the handle_info
clause now names the landmine: every exit signal stops the server, `:normal`
included, and `Generator` used to leave exactly such a signal behind because an
already-queued `{:EXIT, pid, :normal}` survives `Process.unlink/1`. A server
driving a generator would have shut down with reason `:normal` and no log line.
@nyo16 nyo16 changed the title Review remediation Review remediation: close all 3 blockers and 27 warnings/suggestions Jul 27, 2026
@nyo16
nyo16 merged commit 4ed2647 into master Jul 27, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant