Skip to content

feat(wiki-engine): add WASM tree-sitter AST track for code knowledge graph - #304

Open
m0Nst3r873 wants to merge 5 commits into
Tencent:mainfrom
m0Nst3r873:feature/ast-code-knowledge
Open

feat(wiki-engine): add WASM tree-sitter AST track for code knowledge graph#304
m0Nst3r873 wants to merge 5 commits into
Tencent:mainfrom
m0Nst3r873:feature/ast-code-knowledge

Conversation

@m0Nst3r873

Copy link
Copy Markdown
Collaborator

What & Why

The code knowledge graph extractors (src/wiki-engine/code-knowledge/) were purely regex / line-based. That has two concrete problems:

  1. False-positive edges. Dependency edges were built by path-substring matching (file.includes(importPath)), so an import "./user" wrongly links to both user.ts and user-repo.ts (any path containing user).
  2. No call relationships. Only DEPENDS_ON (imports) existed; call sites were invisible.

This PR adds a real AST track using web-tree-sitter (pure-WASM, no native toolchain), ported from the team-wiki reference implementation, for TypeScript/JavaScript, Python, and Go. It runs alongside the existing regex heuristic track (which still covers Java/Rust/config), and AST results win on merge.

How it works

  • New ast/ module: WASM parser registry (async one-time init), tree-sitter queries, symbol/import/call-site walk, import & call resolvers, and fact/edge adapters. Emits precise file-to-file DEPENDS_ON / REFERENCES edges tagged source: "code-ast" with confidence weights.
  • Dual-track with graceful fallback: if the WASM runtime can't load, or TEAMAI_SKIP_AST=1 is set, extraction falls back to heuristic-only and records an AST_UNAVAILABLE gap.
  • code-graph: AST relation facts build precise edges directly, instead of being re-fuzzed through the path-substring matcher.
  • enrich: manifest edges now preserve real AST relation/source provenance (deterministic rank-based merge) instead of hardcoding DEPENDS_ON / code-heuristic. Combined with upstream's resolveImportToModule resolver.
  • Pinned versions: web-tree-sitter@0.25.10 + tree-sitter-wasms@0.1.13 — the only ABI-14-compatible pair (0.26 rejects these grammars). Both are pure-JS deps resolved from node_modules at runtime; no .wasm files are bundled into dist/.

Accuracy (measured)

On a fixture with a naming-collision decoy (user.ts real target, user-repo.ts decoy that nobody imports):

Mode Edges False positives Precision
Regex only (TEAMAI_SKIP_AST=1) 3 1 (→ user-repo.ts) 67%
AST enabled 2 0 100%

Test Plan

  • npx tsc --noEmit — clean
  • npx vitest run — 2074/2074 pass (includes new src/__tests__/ast-extract.test.ts, 11 cases covering TS/Python/Go extraction, merge precedence, gap recording, and enrich provenance)
  • npm run build — success
  • E2E: real teamai codebase --extract on a multi-language sample repo produces code-ast DEPENDS_ON + REFERENCES edges; TEAMAI_SKIP_AST=1 falls back to heuristic-only + writes AST_UNAVAILABLE gap
  • Packaged form: npm pack + install into a clean consumer — all .wasm files resolve from node_modules, AST track engages via the installed CLI (verifies npx/npm distribution works)

Docs

README and usage-guide updated in both EN and zh-CN; new TEAMAI_SKIP_AST env var documented.

Compatibility notes for downstream consumers

  • recall graph-boost already registers REFERENCES in RELATION_WEIGHT and uses maxBoost semantics, so the new edges participate correctly with no change and can't inflate scores via duplicate edges.
  • GraphEdgeSource already contains code-ast; no schema change.

🤖 Generated with Claude Code

m0Nst3r873 and others added 2 commits August 20, 2026 20:40
…graph

The code knowledge graph extractors were purely regex/line-based, which
produced false-positive dependency edges (path-substring matching) and had
no notion of call relationships. This adds a real AST track using
web-tree-sitter (pure-WASM, no native toolchain) for TypeScript/JavaScript,
Python, and Go, ported from the team-wiki reference implementation.

- New ast/ module: WASM parser registry (async one-time init), tree-sitter
  queries, symbol/import/call-site walk, import & call resolvers, and
  fact/edge adapters. Emits precise file-to-file DEPENDS_ON / REFERENCES
  edges tagged source:"code-ast" with confidence weights.
- Dual-track: runs alongside the regex heuristic track (which still covers
  Java/Rust/config); AST facts win on merge. Falls back to heuristic-only
  and records an AST_UNAVAILABLE gap when the runtime is unavailable or
  TEAMAI_SKIP_AST=1.
- code-graph: AST relation facts build precise edges instead of being
  re-fuzzed through the path-substring matcher.
- enrich: manifest edges now preserve real AST relation/source provenance
  (deterministic rank-based merge) instead of hardcoding DEPENDS_ON /
  code-heuristic.
- Pinned web-tree-sitter@0.25.10 + tree-sitter-wasms@0.1.13 (only ABI-14
  compatible pair; 0.26 rejects these grammars).
- Docs (README + usage-guide, EN/zh-CN) and unit tests added.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- walk.ts: free the parsed tree-sitter Tree via try/finally { tree.delete() }
  to release WASM off-heap memory; JS GC does not reclaim it, so large repos
  would otherwise grow the emscripten heap unbounded.
- call-resolver.ts: memoize import bindings per source file in resolveCallSites
  (was rebuilt for every call site — O(callSites × imports)).
- import-bindings.ts: detect Python exports via module-level class/function
  definitions instead of the non-existent "__export__" node type, so Python
  symbol-level call resolution works (was always exported=false).
- merge-edges.ts / import-resolver.ts: drop unused findConflictingEdges,
  EdgeConflict, and clearTsconfigCache (speculative dead code).
- walk.ts: simplify a no-op ternary on the call receiver.
- tests: add a Python module-level-export regression case.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@m0Nst3r873

Copy link
Copy Markdown
Collaborator Author

Code review follow-up (pushed in 6555763)

Ran a senior-engineer review pass over the diff. No P0. Addressed the following in this branch:

Correctness / robustness (fixed):

  • WASM memory leakwalkFile now frees the parsed tree-sitter Tree via try/finally { tree.delete() }. JS GC does not reclaim WASM off-heap memory, so a large repo would have grown the emscripten heap unbounded.
  • Quadratic call resolutionresolveCallSites now memoizes import bindings per source file (was rebuilt for every call site).
  • Python export detection — was gated on a non-existent __export__ tree-sitter node, so every Python symbol was exported=false and symbol-level call resolution never fired. Now detects module-level class/def. Added a regression test.

Cleanup (fixed):

  • Removed speculative dead code (findConflictingEdges, EdgeConflict, clearTsconfigCache) and simplified a no-op ternary.

Known limitations (deliberately deferred — the regex heuristic track still fills these gaps)

These are AST-resolver precision gaps; when the AST track doesn't resolve an edge it degrades to the heuristic track, so no coverage is lost — only precision is bounded:

  • Python multi-segment imports (from a.b.c import x) don't map dotted paths to nested files yet — only single-segment sibling imports resolve.
  • Go cross-package imports using full module paths (github.com/org/repo/pkg) resolve to an external/unresolved gap (no go.mod module-prefix stripping).
  • IMPLEMENTS edges are handled downstream but no query capture emits them yet.

Happy to fold any of these into this PR if preferred, or track them as follow-ups.

Full suite green after the fixes: tsc clean, 2075 tests pass, build + real-CLI E2E verified.

Two AST-resolver precision improvements from PR review follow-up:

- Python multi-segment imports: `from a.b.c import x` now maps the dotted
  module to a nested path (a/b/c.py or a/b/c/__init__.py) instead of only
  resolving single-segment sibling imports.
- TS/TSX IMPLEMENTS edges: a class's `implements` clause now produces
  IMPLEMENTS edges (source:"code-ast") to the interface's defining file,
  resolved via same-file interface symbols or imported bindings. A separate
  query pattern avoids the capture-map collision when a class implements
  multiple interfaces; names that resolve to neither (ambient/global types)
  are skipped rather than emitting a spurious edge.

Adds tests for both; Go cross-package module resolution remains a known
follow-up.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@m0Nst3r873

Copy link
Copy Markdown
Collaborator Author

Addressed two of the three known limitations (pushed in 3add345)

  • Python multi-segment importsfrom a.b.c import x now maps the dotted module to a nested path (a/b/c.py or a/b/c/__init__.py). Previously only single-segment sibling imports resolved.
  • TS/TSX IMPLEMENTS edges — a class's implements clause now emits IMPLEMENTS edges (source: "code-ast") to the interface's defining file, resolved via same-file interface symbols or imported bindings. Uses a separate query pattern so a class implementing multiple interfaces doesn't collide in the capture map; names that resolve to neither (ambient/global types) are skipped rather than emitting a spurious edge.

Both covered by new tests (unit + real-CLI E2E). Full suite: tsc clean, 2078 tests pass.

Still deferred: Go cross-package imports (github.com/org/repo/pkg) — needs go.mod module-prefix stripping plus directory-level package resolution; lower value / higher cost, and the heuristic track still covers it. Happy to take it in a follow-up if desired.

m0Nst3r873 and others added 2 commits August 21, 2026 10:41
Keep README and usage-guide (EN/zh-CN) in sync with the new TS implements
edge support.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ping AST edges

Batch imports (--from-repo-list and --from-org, which run importFromRepo
concurrently at concurrency>=2) were losing AST edges: the final per-repo
graph-index.json committed to the team repo contained only code-heuristic
edges, while single --from-repo and concurrency=1 produced the correct
code-ast edges.

Root cause: each concurrent importFromRepo writes into the shared
teamwikiRoot (per-repo graph copy, facts/interfaces caches, source-manifest,
router/index, and reconcileKnowledge's global graph). Those are
read-modify-write operations on shared files, so parallel repos clobbered
each other's artifacts.

Fix: a module-level promise-chain mutex. The clone + extractCodebase phase
(the expensive part, which writes only to the per-repo cache dir) stays
parallel; only the team-repo write phase is serialized. A repo acquires the
lock after extract and releases it in the existing finally, so it is
exception-safe and never deadlocks. Verified: with the fix, concurrency=3
batch import produces the same code-ast edge counts as concurrency=1.

Adds unit tests asserting the mutex's mutual-exclusion, re-acquire, and FIFO
contract.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant