Perf/llama packed load memory#221
Closed
michalharakal wants to merge 2 commits into
Closed
Conversation
…ak RSS The NATIVE_OPTIMIZED Llama load peaks ~3.2 GB for a 668 MB model and OOM-kills the 2 GB board. Two causes addressed: - Fused load+pack: pack each raw-quant tensor inline in the streaming GGUF loader (DecoderGgufWeightLoader) instead of loading all tensors then converting in a second pass. convertLlamaWeightsPacked is now idempotent (empty quantTypes) and drains its source map destructively. Shared packLlamaTensor() helper factors the pack/dequant decision. - Per-tensor gcCollectHint() (expect/actual across all targets; GC.collect on K/Native, no-op elsewhere) bounds the transient-copy high-water to ~one tensor in flight, mirroring the GemmaDecoder board pattern. Compiles across all targets (jvm/native/js/wasm/android); publishToMavenLocal green. Host/board RSS measurement pending a working native consume path. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sibling of core's KernelProfile (matmul dispatch): accumulates wall time per named phase across the eager decode path — attn qkv/o projections, reshape, RoPE, KV-cache update, fused-attention copy vs compute, RMSNorm, SwiGLU proj vs eltwise, lm_head, residual, sampling. Diagnostic-only, always-on counters; report/reset driven by the consumer (tinyllama SKAINET_PROFILE gate). Needed to rank the ~54% non-matmul tail measured on the SL2619 board before fixing it. Host-validated (macosArm64 eager, 8 tok): output unchanged and correct, buckets sum to the measured inference time. Perf-Tag: board-tail-profile Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
Author
|
Superseded by #220: this branch is the pre-rebase (0.32.1-based) version of the same two commits (fused load+pack b5dbb47 / 1107fc9, PhaseProfile 7c8e4c8 / 58982d7). They were rebased onto 0.34.0 as perf/llama-packed-load-memory-0.34 — resolving the VoidDense.kt addBias conflict with the Moonshine work — and merged into develop via #220. Both changes are verified present on develop; nothing from this branch is lost. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.