fix(llama-cpp): explain tensor count mismatch - #11597
Open
localai-org-maint-bot wants to merge 2 commits into
Open
fix(llama-cpp): explain tensor count mismatch#11597localai-org-maint-bot wants to merge 2 commits into
localai-org-maint-bot wants to merge 2 commits into
Conversation
llama.cpp reports the same tensor-count error for unsupported model layouts and damaged GGUF files. Add a focused hint so operators can update the backend or verify the model without losing the upstream diagnostic. Assisted-by: Codex:gpt-5.6
The generated gRPC source tree omitted the new header and test. Every llama.cpp-derived backend therefore failed when grpc-server.cpp included the missing header. Assisted-by: Codex:gpt-5.6 [systematic-debugging]
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR preserves llama.cpp tensor-count load errors and appends an actionable hint that the GGUF layout may be incompatible with the selected backend or the file may be corrupt. It leaves unrelated load errors unchanged.
Ollama-published monolithic Gemma 4 blobs use Ollama's private compatibility layer to hide embedded projector tensors. LocalAI uses upstream llama.cpp, so its loader sees the extra tensors and rejects that internal blob layout. The model-scoped backend process then exits after failed initialization; LocalAI itself remains available.
Closes #11593
Verification
The standalone regression test covers the exact
expected 2131, got 720diagnostic, a generic tensor-count mismatch, and an unrelated load error.Notes for Reviewers
The change deliberately does not import Ollama's compatibility patches, pre-parse GGUF metadata, or alter RPC status and backend lifecycle.
Signed commits