Background
Currently akb embed only produces chunk-level embeddings (chunks.embedding). The kb-viz SemanticFrame shows "No embeddings at level 'document'" when viewed at the document or expression level, because only chunks have embedding vectors in the manifest.
Goal
Enable the SemanticFrame (UMAP scatter) and GraphFrame (similarity edges) to show meaningful layouts at all three node levels:
| Level |
Embedding source |
document |
Mean-pool of all its chunk embeddings |
chunk |
Already exists — chunks.embedding |
expression |
Embed the NER span's raw_text directly (short text, fast) |
Proposed changes
cli/embed.py
- Add
--level document / --level expression / --level all flags
- Document: after chunk embedding, compute
mean(chunk_vecs) per block and store in a new blocks.embedding column (BLOB, same format as chunks.embedding)
- Expression: embed
ner_spans.raw_text and store in a new ner_spans.embedding column
cli/db.py
- Add
update_block_embedding(conn, block_id, blob, run_id)
- Add
update_span_embedding(conn, span_id, blob, run_id)
- Migration helper:
ALTER TABLE blocks ADD COLUMN embedding BLOB / ALTER TABLE ner_spans ADD COLUMN embedding BLOB (no-op if column exists)
kb_viz/akb_adapter.py
- Read
blocks.embedding → Node.embedding for document nodes
- Read
ner_spans.embedding → Node.embedding for expression nodes
kb-viz frontend (follow-up)
- SemanticFrame already reads
node.embedding — no change needed once the adapter emits the vectors
- Consider adding similarity edges (type
similarity) between document nodes based on cosine distance, enabling a document-level GraphFrame view
Acceptance criteria
Background
Currently
akb embedonly produces chunk-level embeddings (chunks.embedding). The kb-viz SemanticFrame shows "No embeddings at level 'document'" when viewed at the document or expression level, because only chunks have embedding vectors in the manifest.Goal
Enable the SemanticFrame (UMAP scatter) and GraphFrame (similarity edges) to show meaningful layouts at all three node levels:
documentchunkchunks.embeddingexpressionraw_textdirectly (short text, fast)Proposed changes
cli/embed.py--level document/--level expression/--level allflagsmean(chunk_vecs)per block and store in a newblocks.embeddingcolumn (BLOB, same format aschunks.embedding)ner_spans.raw_textand store in a newner_spans.embeddingcolumncli/db.pyupdate_block_embedding(conn, block_id, blob, run_id)update_span_embedding(conn, span_id, blob, run_id)ALTER TABLE blocks ADD COLUMN embedding BLOB/ALTER TABLE ner_spans ADD COLUMN embedding BLOB(no-op if column exists)kb_viz/akb_adapter.pyblocks.embedding→Node.embeddingfor document nodesner_spans.embedding→Node.embeddingfor expression nodeskb-viz frontend (follow-up)
node.embedding— no change needed once the adapter emits the vectorssimilarity) between document nodes based on cosine distance, enabling a document-level GraphFrame viewAcceptance criteria
akb embed --all --level documentcomputes and stores block-level embeddingsakb embed --all --level expressionembeds span textsakb embed --all --level alldoes all three levels in one passakb_adapter.pyexports embeddings for all three node typesdocumentlevel when data is loadedexpressionlevel