[codex] Add SSD-aware GGUF mmap policy

[Jackson57279]

Summary

• Add GGUF mmap load policies: demand, prefetch, sequential, and random.
• Expose --mmap-policy in oxidize-cpp and validate it before model loading or --print-plan.
• Extend autotune output with mmap_policy, choosing demand for huge/over-RAM models and prefetch otherwise.

Research basis

• FlexGen motivates explicit tensor placement and CPU/disk offload under constrained memory.
• LLM in a Flash motivates demand loading from flash and avoiding unnecessary DRAM transfer.
• ZeRO-Inference motivates treating CPU/NVMe as part of the inference memory hierarchy.
• Linux madvise documents the paging hints used by the implementation.

Validation

• RED before implementation: .omo/ulw-loop/019f2383-7472-7f40-9115-29b70a98d721/evidence/c001-red.txt, c002-red.txt.
• GREEN tests: ./build/autotune_test, ./build/gguf_mmap_policy_test.
• Manual QA: tmux --print-plan for a 250G sparse model produced "mmap_policy": "demand".
• Manual QA: invalid --mmap-policy nonsense exited with code 2.
• Remote QA: built and tested on ai@192.168.1.132 in a temp copy; autotune_test, gguf_mmap_policy_test, and oxidize-cpp --help all passed.
• Clean staged-patch check: temporary git worktree applied the staged diff and passed the focused build/tests.

Note

Full local cmake --build build is currently blocked by an unrelated untracked tests/train_matmul_test.cpp that references missing training symbols; the PR excludes that file.

---

Summary by cubic

Adds SSD-aware GGUF mmap policies and a --mmap-policy flag, with autotune choosing demand paging for huge/over-RAM models to avoid full-file prefetch and speed up startup/IO. Also updates Rust anyhow to 1.0.103 to address a security advisory.

• New Features

  • GGUF mmap policies: demand, prefetch, sequential, random (default: prefetch).
  • New CLI flag: --mmap-policy <mode> with validation; included in --print-plan JSON as mmap_policy.
  • GgufModel::load accepts GgufLoadOptions and applies madvise per policy, including split shards.
  • Autotune sets mmap_policy (demand for huge models, prefetch otherwise); CLI respects explicit overrides and shows it in the plan.

• Dependencies

  • Bumped anyhow to 1.0.103 (fixes RUSTSEC-2026-0190).

^{Written for commit 808e101. Summary will update on new commits.}

[cubic-dev-ai]

2 issues found across 7 files

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cursor]

Bugbot is not enabled for your account, so this pull request was not reviewed.

Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs.

[Jackson57279]

Addressed the cubic.dev review in b168577:

• P1 (asserts disabled under NDEBUG): replaced all assert() calls in gguf_mmap_policy_test.cpp (and the remaining ones in autotune_test.cpp, which had the same issue) with an abort-based require() helper that works regardless of build type.
• P2 (missing policy coverage): the round-trip test now covers all four policies — demand, prefetch, sequential, random — plus an empty-string rejection case.

Verified with a Release (-DNDEBUG) build: gguf_mmap_policy_test, autotune_test, and the rest of the ctest suite pass.

[cubic-dev-ai]

1 issue found across 2 files (changes from recent commits).

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="oxidize-cpp/tests/gguf_mmap_policy_test.cpp">

<violation number="1" location="oxidize-cpp/tests/gguf_mmap_policy_test.cpp:25">
P3: check_round_trip uses the policy name as the error message for all three require() calls, so a failure at any step produces the same output without indicating which check broke (parse existence, value equality, or name round-trip). Consider using distinct messages per check so the failing assertion is immediately identifiable from stderr.</violation>
</file>

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

P3: check_round_trip uses the policy name as the error message for all three require() calls, so a failure at any step produces the same output without indicating which check broke (parse existence, value equality, or name round-trip). Consider using distinct messages per check so the failing assertion is immediately identifiable from stderr.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At oxidize-cpp/tests/gguf_mmap_policy_test.cpp, line 25:

<comment>check_round_trip uses the policy name as the error message for all three require() calls, so a failure at any step produces the same output without indicating which check broke (parse existence, value equality, or name round-trip). Consider using distinct messages per check so the failing assertion is immediately identifiable from stderr.</comment>

<file context>
@@ -1,24 +1,35 @@
+  auto parsed = oxidize::parse_mmap_policy(name);
+  require(parsed.has_value(), name);
+  require(*parsed == expected, name);
+  require(std::string(oxidize::mmap_policy_name(*parsed)) == name, name);
+}
 
</file context>

[Jackson57279]

Triage note (maintainer pass): mmap-policy work looks clean and all CI is green, and the earlier cubic P1/P2 review was addressed.

The main thing to resolve before merge is ordering vs the stack: #29 builds directly on this branch, so merging #29 pulls #28's commits in with it. To avoid a redundant merge, either land this one first and then rebase #29 onto master, or fold #28 into #29 and close this. Please confirm which path you intend.

Also note the anyhow 1.0.103 bump (RUSTSEC-2026-0190) appears in several open PRs (#28/#32/#33/#34) — whichever lands first resolves it; the rest should drop it on rebase to keep diffs clean.