feat(wiki-engine): add WASM tree-sitter AST track for code knowledge graph - #304
feat(wiki-engine): add WASM tree-sitter AST track for code knowledge graph#304m0Nst3r873 wants to merge 5 commits into
Conversation
…graph The code knowledge graph extractors were purely regex/line-based, which produced false-positive dependency edges (path-substring matching) and had no notion of call relationships. This adds a real AST track using web-tree-sitter (pure-WASM, no native toolchain) for TypeScript/JavaScript, Python, and Go, ported from the team-wiki reference implementation. - New ast/ module: WASM parser registry (async one-time init), tree-sitter queries, symbol/import/call-site walk, import & call resolvers, and fact/edge adapters. Emits precise file-to-file DEPENDS_ON / REFERENCES edges tagged source:"code-ast" with confidence weights. - Dual-track: runs alongside the regex heuristic track (which still covers Java/Rust/config); AST facts win on merge. Falls back to heuristic-only and records an AST_UNAVAILABLE gap when the runtime is unavailable or TEAMAI_SKIP_AST=1. - code-graph: AST relation facts build precise edges instead of being re-fuzzed through the path-substring matcher. - enrich: manifest edges now preserve real AST relation/source provenance (deterministic rank-based merge) instead of hardcoding DEPENDS_ON / code-heuristic. - Pinned web-tree-sitter@0.25.10 + tree-sitter-wasms@0.1.13 (only ABI-14 compatible pair; 0.26 rejects these grammars). - Docs (README + usage-guide, EN/zh-CN) and unit tests added. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- walk.ts: free the parsed tree-sitter Tree via try/finally { tree.delete() }
to release WASM off-heap memory; JS GC does not reclaim it, so large repos
would otherwise grow the emscripten heap unbounded.
- call-resolver.ts: memoize import bindings per source file in resolveCallSites
(was rebuilt for every call site — O(callSites × imports)).
- import-bindings.ts: detect Python exports via module-level class/function
definitions instead of the non-existent "__export__" node type, so Python
symbol-level call resolution works (was always exported=false).
- merge-edges.ts / import-resolver.ts: drop unused findConflictingEdges,
EdgeConflict, and clearTsconfigCache (speculative dead code).
- walk.ts: simplify a no-op ternary on the call receiver.
- tests: add a Python module-level-export regression case.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Code review follow-up (pushed in 6555763)Ran a senior-engineer review pass over the diff. No P0. Addressed the following in this branch: Correctness / robustness (fixed):
Cleanup (fixed):
Known limitations (deliberately deferred — the regex heuristic track still fills these gaps)These are AST-resolver precision gaps; when the AST track doesn't resolve an edge it degrades to the heuristic track, so no coverage is lost — only precision is bounded:
Happy to fold any of these into this PR if preferred, or track them as follow-ups. Full suite green after the fixes: |
Two AST-resolver precision improvements from PR review follow-up: - Python multi-segment imports: `from a.b.c import x` now maps the dotted module to a nested path (a/b/c.py or a/b/c/__init__.py) instead of only resolving single-segment sibling imports. - TS/TSX IMPLEMENTS edges: a class's `implements` clause now produces IMPLEMENTS edges (source:"code-ast") to the interface's defining file, resolved via same-file interface symbols or imported bindings. A separate query pattern avoids the capture-map collision when a class implements multiple interfaces; names that resolve to neither (ambient/global types) are skipped rather than emitting a spurious edge. Adds tests for both; Go cross-package module resolution remains a known follow-up. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Addressed two of the three known limitations (pushed in 3add345)
Both covered by new tests (unit + real-CLI E2E). Full suite: Still deferred: Go cross-package imports ( |
Keep README and usage-guide (EN/zh-CN) in sync with the new TS implements edge support. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ping AST edges Batch imports (--from-repo-list and --from-org, which run importFromRepo concurrently at concurrency>=2) were losing AST edges: the final per-repo graph-index.json committed to the team repo contained only code-heuristic edges, while single --from-repo and concurrency=1 produced the correct code-ast edges. Root cause: each concurrent importFromRepo writes into the shared teamwikiRoot (per-repo graph copy, facts/interfaces caches, source-manifest, router/index, and reconcileKnowledge's global graph). Those are read-modify-write operations on shared files, so parallel repos clobbered each other's artifacts. Fix: a module-level promise-chain mutex. The clone + extractCodebase phase (the expensive part, which writes only to the per-repo cache dir) stays parallel; only the team-repo write phase is serialized. A repo acquires the lock after extract and releases it in the existing finally, so it is exception-safe and never deadlocks. Verified: with the fix, concurrency=3 batch import produces the same code-ast edge counts as concurrency=1. Adds unit tests asserting the mutex's mutual-exclusion, re-acquire, and FIFO contract. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
What & Why
The code knowledge graph extractors (
src/wiki-engine/code-knowledge/) were purely regex / line-based. That has two concrete problems:file.includes(importPath)), so animport "./user"wrongly links to bothuser.tsanduser-repo.ts(any path containinguser).DEPENDS_ON(imports) existed; call sites were invisible.This PR adds a real AST track using
web-tree-sitter(pure-WASM, no native toolchain), ported from the team-wiki reference implementation, for TypeScript/JavaScript, Python, and Go. It runs alongside the existing regex heuristic track (which still covers Java/Rust/config), and AST results win on merge.How it works
ast/module: WASM parser registry (async one-time init), tree-sitter queries, symbol/import/call-site walk, import & call resolvers, and fact/edge adapters. Emits precise file-to-fileDEPENDS_ON/REFERENCESedges taggedsource: "code-ast"with confidence weights.TEAMAI_SKIP_AST=1is set, extraction falls back to heuristic-only and records anAST_UNAVAILABLEgap.code-graph: AST relation facts build precise edges directly, instead of being re-fuzzed through the path-substring matcher.enrich: manifest edges now preserve real AST relation/source provenance (deterministic rank-based merge) instead of hardcodingDEPENDS_ON/code-heuristic. Combined with upstream'sresolveImportToModuleresolver.web-tree-sitter@0.25.10+tree-sitter-wasms@0.1.13— the only ABI-14-compatible pair (0.26 rejects these grammars). Both are pure-JS deps resolved fromnode_modulesat runtime; no.wasmfiles are bundled intodist/.Accuracy (measured)
On a fixture with a naming-collision decoy (
user.tsreal target,user-repo.tsdecoy that nobody imports):TEAMAI_SKIP_AST=1)user-repo.ts)Test Plan
npx tsc --noEmit— cleannpx vitest run— 2074/2074 pass (includes newsrc/__tests__/ast-extract.test.ts, 11 cases covering TS/Python/Go extraction, merge precedence, gap recording, and enrich provenance)npm run build— successteamai codebase --extracton a multi-language sample repo producescode-astDEPENDS_ON+REFERENCESedges;TEAMAI_SKIP_AST=1falls back to heuristic-only + writesAST_UNAVAILABLEgapnpm pack+ install into a clean consumer — all.wasmfiles resolve fromnode_modules, AST track engages via the installed CLI (verifiesnpx/npm distribution works)Docs
README and usage-guide updated in both EN and zh-CN; new
TEAMAI_SKIP_ASTenv var documented.Compatibility notes for downstream consumers
recallgraph-boost already registersREFERENCESinRELATION_WEIGHTand usesmaxBoostsemantics, so the new edges participate correctly with no change and can't inflate scores via duplicate edges.GraphEdgeSourcealready containscode-ast; no schema change.🤖 Generated with Claude Code