Skip to content

feat(embed): generate document- and expression-level embeddings for multi-level semantic views #55

Description

@Korede-TA

Background

Currently akb embed only produces chunk-level embeddings (chunks.embedding). The kb-viz SemanticFrame shows "No embeddings at level 'document'" when viewed at the document or expression level, because only chunks have embedding vectors in the manifest.

Goal

Enable the SemanticFrame (UMAP scatter) and GraphFrame (similarity edges) to show meaningful layouts at all three node levels:

Level Embedding source
document Mean-pool of all its chunk embeddings
chunk Already exists — chunks.embedding
expression Embed the NER span's raw_text directly (short text, fast)

Proposed changes

cli/embed.py

  • Add --level document / --level expression / --level all flags
  • Document: after chunk embedding, compute mean(chunk_vecs) per block and store in a new blocks.embedding column (BLOB, same format as chunks.embedding)
  • Expression: embed ner_spans.raw_text and store in a new ner_spans.embedding column

cli/db.py

  • Add update_block_embedding(conn, block_id, blob, run_id)
  • Add update_span_embedding(conn, span_id, blob, run_id)
  • Migration helper: ALTER TABLE blocks ADD COLUMN embedding BLOB / ALTER TABLE ner_spans ADD COLUMN embedding BLOB (no-op if column exists)

kb_viz/akb_adapter.py

  • Read blocks.embeddingNode.embedding for document nodes
  • Read ner_spans.embeddingNode.embedding for expression nodes

kb-viz frontend (follow-up)

  • SemanticFrame already reads node.embedding — no change needed once the adapter emits the vectors
  • Consider adding similarity edges (type similarity) between document nodes based on cosine distance, enabling a document-level GraphFrame view

Acceptance criteria

  • akb embed --all --level document computes and stores block-level embeddings
  • akb embed --all --level expression embeds span texts
  • akb embed --all --level all does all three levels in one pass
  • akb_adapter.py exports embeddings for all three node types
  • SemanticFrame shows a UMAP plot at the document level when data is loaded
  • SemanticFrame shows a UMAP plot at the expression level

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions