embed-gemma:300m returns a collapsed embedding space: deterministic, but unusable for retrieval (same model via llama.cpp is correct)

[teconomix]

Summary

embed-gemma:300m on the FLM backend returns deterministic but semantically collapsed embeddings. Vectors are stable across repeated requests, so this is not the instability from #647 — but the space they live in carries almost no semantic signal, which makes the endpoint unusable for retrieval.

The clearest symptom: for the query "What is the capital of France?", the sentence "Bananas are a yellow tropical fruit." scores higher than "Paris is the capital city of France."

The same model file, same server, same API layer, same client — but served through llama.cpp instead of FLM — behaves correctly. That isolates the difference to the FLM embedding path rather than to the model, the client, or the surrounding server.

	FLM (embed-gemma:300m, q4)	llama.cpp (embeddinggemma-300M-Q8_0.gguf)
cos(query, Paris…)	0.8455	0.8745
cos(query, Bananas…)	0.8684 ← wins	0.2737
pairwise cos, 5 unrelated sentences (median)	0.849	0.245
cos(A,B) same ending, unrelated content	0.8449	0.3900
cos(A,C) same content, different ending	0.9835	0.7598
identical input → distinct vectors (10 runs)	1	1

A pairwise cosine of 0.849 between five completely unrelated sentences is the core of it: almost everything is close to everything, so ranking degenerates to noise.

On a real retrieval benchmark (346 chunks of German technical documentation, 14 questions with a known gold chunk) FLM scored R@5 = 0.00, with the gold chunk at a median rank of 24.5 out of 50 candidates — the expected value for random ordering is 25.5. The same corpus and questions against the llama.cpp build of the same model: R@5 = 0.64.

Environment

FLM version	v0.9.45
Embedding model	embed-gemma:300m (q4, per the model docs)
Served through	Lemonade Server 11.5.0, flm recipe (see isolation note below)
Endpoint	POST /v1/embeddings
CPU / NPU	AMD Ryzen AI 9 HX PRO 370 (Strix Point), XDNA2
OS	Debian 13 (trixie), unprivileged LXC on Proxmox, kernel 7.0.14
Control group	llama.cpp (Vulkan) serving ggml-org/embeddinggemma-300M-GGUF:embeddinggemma-300M-Q8_0.gguf

Isolation note: I drive FLM through Lemonade's flm recipe rather than bare flm serve. The control group runs through the same Lemonade instance, the same /v1/embeddings route and the same client — only the backend differs. Everything above the backend is therefore held constant. I have not reproduced against bare flm serve; if that matters for triage I am happy to.

Reproduction

stdlib-only, no dependencies:

#!/usr/bin/env python3
"""usage: python3 repro.py [BASE_URL] [MODEL]"""
import itertools, json, math, statistics, sys, urllib.request

BASE  = sys.argv[1] if len(sys.argv) > 1 else "http://127.0.0.1:52625/v1"
MODEL = sys.argv[2] if len(sys.argv) > 2 else "embed-gemma:300m"

def embed(texts):
    req = urllib.request.Request(
        BASE.rstrip("/") + "/embeddings",
        data=json.dumps({"model": MODEL, "input": texts}).encode(),
        headers={"Content-Type": "application/json"})
    r = json.loads(urllib.request.urlopen(req, timeout=300).read())
    return [d["embedding"] for d in sorted(r["data"], key=lambda x: x["index"])]

def cos(a, b):
    return (sum(x*y for x, y in zip(a, b))
            / ((math.sqrt(sum(x*x for x in a)) or 1e-9)
               * (math.sqrt(sum(x*x for x in b)) or 1e-9)))

q = "What is the capital of France?"
docs = ["Paris is the capital city of France.",
        "Bananas are a yellow tropical fruit.",
        "The engine requires regular oil changes."]
v = embed([q] + docs)
print("1) TRIVIAL RANKING   query: %s" % q)
for s, d in sorted([(cos(v[0], v[i+1]), docs[i]) for i in range(3)], reverse=True):
    print("   %+.4f  %s" % (s, d))

unrelated = ["The dog barks loudly in the garden.",
             "Quantum physics describes subatomic particles.",
             "I buy bread at the bakery every morning.",
             "The stock market closed lower today.",
             "Please take out the rubbish tonight."]
uv = embed(unrelated)
pairs = [cos(a, b) for a, b in itertools.combinations(uv, 2)]
print("2) ANISOTROPY   median %.3f   min %.3f   max %.3f"
      % (statistics.median(pairs), min(pairs), max(pairs)))

a = "The dog barks loudly in the garden. End."
b = "Quantum physics describes subatomic particles. End."
c = "The dog barks loudly in the garden. Beginning."
w = embed([a, b, c])
print("3) CONTENT SENSITIVITY")
print("   cos(A,B)  same ending, unrelated content : %.4f" % cos(w[0], w[1]))
print("   cos(A,C)  same content, different ending : %.4f" % cos(w[0], w[2]))

import hashlib
h = {}
for _ in range(10):
    k = hashlib.sha1(",".join(map(str, embed([q])[0])).encode()).hexdigest()[:12]
    h[k] = h.get(k, 0) + 1
print("4) DETERMINISM   10x same input -> %d distinct vector(s) %s" % (len(h), h))

Actual output — FLM

1) TRIVIAL RANKING   query: What is the capital of France?
   +0.8684  Bananas are a yellow tropical fruit.
   +0.8455  Paris is the capital city of France.
   +0.7803  The engine requires regular oil changes.
2) ANISOTROPY   median 0.849   min 0.798   max 0.944
3) CONTENT SENSITIVITY
   cos(A,B)  same ending, unrelated content : 0.8449
   cos(A,C)  same content, different ending : 0.9835
4) DETERMINISM   10x same input -> 1 distinct vector(s) {'9533a954a5ee': 10}

Actual output — llama.cpp, same model, same server

1) TRIVIAL RANKING   query: What is the capital of France?
   +0.8745  Paris is the capital city of France.
   +0.2737  Bananas are a yellow tropical fruit.
   +0.2008  The engine requires regular oil changes.
2) ANISOTROPY   median 0.245   min 0.177   max 0.343
3) CONTENT SENSITIVITY
   cos(A,B)  same ending, unrelated content : 0.3900
   cos(A,C)  same content, different ending : 0.7598
4) DETERMINISM   10x same input -> 1 distinct vector(s) {'52707172da1b': 10}

Ruled out

Before filing I tested the obvious configuration causes; none of them explain it:

Hypothesis	Test	Result
Batch reordering / index mismatch	batched vs. one-request-per-string	vectors bit-identical, cos = 1.000000
Missing EmbeddingGemma task prefixes (task: search result | query: / title: none | text:)	trivial ranking with the documented prefixes	slightly worse (Paris 0.757, oil changes 0.877)
Embedding model not co-loaded with an LLM, as the model docs require (flm serve <llm> --embed 1)	loaded qwen3.5-0.8b-FLM onto the NPU alongside it, re-ran	anisotropy 0.783 → 0.782, i.e. unchanged
A constant offset dominating the vectors	subtracted the corpus mean (centering) before ranking	median rank 41 → 61 of 120, i.e. worse, exactly random

Non-determinism is also ruled out for this report: 20 identical requests returned 20 identical vectors on this setup.

Relation to existing issues

• Embeddings are non-deterministic: identical input returns different vectors, and processing a longer document permanently changes the embeddings of shorter ones #647 (non-deterministic embeddings, cross-request contamination) — different symptom, possibly the same root cause. That report is Windows on Ryzen AI 9 HX 370 / FLM v0.9.46; this one is Linux on Ryzen AI 9 HX PRO 370 / FLM v0.9.45, so the same Strix Point family. If the embedding path has shared state or a pooling/normalisation defect, both symptoms could fall out of it.
• embedding model fails #589 ("embedding model fails") — no detail given there; this may or may not be the same thing.

Guess at the cause

I could not narrow it further from the outside. The signature — a stable but collapsed space where content barely moves the vector — usually points at either pooling/normalisation (e.g. wrong token positions being pooled) or quantisation destroying a model too small to absorb it. The model docs list the FLM build as Q4_1, which is a coarse legacy format; a 300M encoder has little redundancy to spare, and the same weights at Q8_0 are fine. Both are consistent with what I measured, and I cannot distinguish them through the API.

Impact

This silently degrades every downstream retrieval system rather than failing loudly. In my case five RAG and agent-memory consumers had been indexing against it for months; nothing errored, results were merely subtly useless, and it only surfaced when I benchmarked retrieval quality directly. A note in the model card would already help; a fix would help more.

Happy to run further tests on this hardware if that is useful.

[Atomic-Germ]

If the cause is the model being q4_1, there's zero chance of this being fixed.