Support Nemotron 3.5 Lightning Q8 GGUF conversion - #497
Open
justinchuby wants to merge 1 commit into
Open
Conversation
6 tasks
Performance Comparison
|
🏗️ Architecture Diff
No architecture changes detected. ✅ Legend: ⚪ No change · 🔵 Minor (attrs/inits) · 🟡 Moderate (nodes added/removed) · 🔴 Major (interface changed) |
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
justinchuby
force-pushed
the
justinchuby-support-nemotron-gguf
branch
2 times, most recently
from
August 15, 2026 06:55
9e6ffbc to
2e31b58
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds a GGUF architecture-adapter framework and a strict Nemotron 3.5 Lightning (nemotron_h_moe) adapter to enable exact Q8_0 preservation during GGUF → ONNX conversion, along with tokenizer reconstruction and direct-ORT validation support.
Changes:
- Introduces a reusable GGUF architecture adapter interface/registry and implements a strict Nemotron adapter that validates the pinned tensor table, schedules, qtypes, and mapping completeness.
- Extends NemotronH model construction to support quantized projections/embeddings (MatMulNBits + GatherBlockQuantized) and adds repacking support for stacked expert tensors.
- Adds strict GPT-2/Pixtral tokenizer reconstruction, broader test coverage for the adapter/tokenizer/repacker paths, and updates docs + an acceptance runner script.
Reviewed changes
Copilot reviewed 17 out of 17 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| src/mobius/models/nemotron_h.py | Adds quantized linear/embedding wiring for NemotronH graphs. |
| src/mobius/models/nemotron_h_test.py | Adds unit coverage for float vs quantized NemotronH graph contracts and initializer coverage validation. |
| src/mobius/integrations/gguf/_tokenizer.py | Adds strict GPT-2/Pixtral tokenizer reconstruction for Nemotron GGUF metadata. |
| src/mobius/integrations/gguf/_tokenizer_test.py | Adds tests for strict Pixtral/GPT-2 reconstruction and pinned-artifact parity hooks. |
| src/mobius/integrations/gguf/_repacker.py | Adds repacking for leading-axis stacked 3D quantized tensors (expert stacks). |
| src/mobius/integrations/gguf/_repacker_test.py | Adds execution-level test validating stacked-expert repack output matches dequantized matmul. |
| src/mobius/integrations/gguf/_nemotron_h_moe.py | New strict Nemotron 3.5 Lightning adapter: validation, config extraction, mapping, and transforms. |
| src/mobius/integrations/gguf/_mmproj_test.py | Updates multimodal preflight expectation to reflect adapter validation errors. |
| src/mobius/integrations/gguf/_config_mapping.py | Allows config extraction via adapter (validated) before generic mapping. |
| src/mobius/integrations/gguf/_builder.py | Routes GGUF build flow through adapters (validation, quant config, mapping-based quant detection/state_dict load). |
| src/mobius/integrations/gguf/_builder_test.py | Updates preflight guard tests now that Nemotron adapter is supported. |
| src/mobius/integrations/gguf/_architecture.py | New adapter registry + mapping audit + package state-dict coverage checker. |
| src/mobius/integrations/gguf/_architecture_test.py | Adds synthetic + optional real-artifact integration validation for the Nemotron adapter. |
| src/mobius/components/_mamba_block.py | Adds linear_class injection point so Mamba blocks can use quantized linears. |
| examples/olive/nemotron-3_5-lightning-30b/validate_gguf_q8.py | Adds a reproducible build/save/reload/direct-ORT acceptance runner for pinned Q8_0. |
| docs/cli_reference.md | Updates CLI docs to reflect supported pinned Nemotron Q8_0 GGUF import. |
| docs/api/build_from_gguf.md | Replaces the prior waiver section with pinned Q8_0 support + reproducibility details. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Add a reusable GGUF architecture-adapter seam and a strict nemotron_h_moe adapter that validates the pinned 52-layer backbone, excludes the auxiliary MTP block, and enforces complete source-to-initializer mapping. Preserve Q8_0 weights in MatMulNBits, including stacked routed experts, reconstruct the pinned Pixtral tokenizer contract, and add a fresh-process direct-ORT acceptance runner with exact llama.cpp generation evidence. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchu@microsoft.com>
justinchuby
force-pushed
the
justinchuby-support-nemotron-gguf
branch
from
August 15, 2026 11:11
2e31b58 to
4e9ab2e
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
nemotron_h_moeowns its schedule, tensor names, transforms, qtype policy, tokenizer contract, and MTP semanticsMatMulNBits(bits=8, block_size=32), including leading-axis routed-expert expansion without dequantizationStack and lineage
This PR is de-stacked on exact final #487 head
a8cd77570bca980861eeadab9e6fa077464f0138. The review-fixed PR head is4e9ab2e51a7ba3d39fa2d54d241be1857f976c0b;a8cd775...HEADcontains exactly one commit,Support Nemotron 3.5 Q8 GGUF conversion. The final #487 skill split removed seven generic skill changes to #500 without changing code/model behavior. No #496 commits or #487 skill changes are duplicated.Exact architecture and mapping contract
The adapter validates the complete pinned tensor table before graph construction:
ssm_a -> log(-A), A/D/grouped-norm squeeze/flatten, conv-axis expansion, inverse llama.cpp Q/K row permutation, and direct expert-axis slicing are shape-validatedAll Nemotron constants and error prose live in
src/mobius/integrations/gguf/_nemotron_h_moe.py; the model/component factory changes are source-format agnostic and the existing float/Safetensors graph remains unchanged. The adapter contract requires every adapter to implement source validation, and generic validation dispatch is explicitly typed.Pinned real-artifact acceptance
Source:
unsloth/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-GGUF@f2d3fe3694501008786e81e5f20360cbf715496aNVIDIA-Nemotron-3.5-Lightning-30B-A3B-Q8_0.ggufdc5276dd0619c04e277504d2358a793e31ccbe39e894d767d0d14f2a221e2ca4The full build/save/reload/prefill/cached-decode acceptance was rerun after the final review fixes on exact head
4e9ab2e51a7bin a clean process:MatMulNBits+ 1GatherBlockQuantizedQuantizeLinearorDequantizeLinearA fresh process loaded the saved external-data package in ONNX Runtime 1.28.0 (
45de2a8b06) onCPUExecutionProvider, ran nonzero unpadded prefill, independently ran right-padded prefill with an explicit mask, and performed every cached decode step:0.0(atol=1e-5)Independent llama.cpp reference: commit
9d57ce456c94d241dde672b2db9cf18879766568, MSVC 19.44.35228.0, CPU only:[1784, 8961, 1307, 5498, 1395][6993, 1046, 1256, 1010, 1784, 8961, 1307, 10787]Paris. \nThe capital of GermanyThe review-fixed exact-head direct-ORT run matched every generated ID and the decoded text exactly.
Tokenizer and special-token contract
The GGUF tokenizer is selected only for exact
model=gpt2, pre=pixtralmetadata and matches the pinned official tokenizer atnvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16@d468880b6ad3c6e0d21377ce7242adaea4cc884dacross vocabulary, added-token flags, pre-tokenizer, decoder, post-processor, encode IDs, and decode text.4999709474e3c967358c1f1199b6be65fb9055d3eb59e0cd387f9e7077fc40edb1b0165185b1925118c2f7b1e978439b02010c3a420ebfec5c19a093a0d9b4cb623c34567aebb18582765289fbe23d901c62704d6518d71866e0e58db892b5b71/11/9991/2/0[2, 11]<|im_end|>ID 11PAD 999 is rejected because it names
<SPECIAL_999>. The GGUF-embedded chat template differs from the pinned official asset, so the recipe verifies immutable official sidecar hashes rather than silently treating it as authoritative. Final #487 sentinel filtering and EOS/EOT/EOM union semantics remain intact for the generic path.Explicit limits
Validation on exact head
1,580 passed, 44 skipped, 176 deselected, 39 xfailed2 passed3,965 passed, 64 skippedgit diff --check: cleana8cd775...4e9ab2e: no significant findingsWaivers