[Bulk index][C/C++] Share one Clang compilation-universe provider

[samchon]

Outcome

Replace the current C/C++ navigation artifact with one shared clangd-owned
semantic provider. The final provider must preserve Clang indexing roles and
relations in incremental background-index shards, treat headers as
translation-unit/configuration views, and export one atomic C/C++ generation
without reparsing every translation unit twice.

Current master baseline

PR #147 registers one
scip-clang provider for C and C++, prepares/validates a compilation database,
tracks generated inputs, and publishes both languages from one session.
Current strict grounding intentionally claims no edge families:
scip-clang 0.4.0 omits the enclosing source information required by the common
adapter to attribute even references safely.

The exact cold results were:

• Redis C: 22.795 s strict versus 262.906 s clangd LSP fallback;
• LevelDB C++: 8.353 s strict versus 26.452 s fallback.

Those fast strict cells are useful navigation baselines, not proof of calls,
reads/writes, construction, inheritance, overrides or complete headers.

The generic/static discovery bug is separately verified on master:
LANGUAGE_SPECS assigns .h only to C and source enumeration for a C++-only
request can omit .h entirely. A request for both languages partitions every
.h into C. Strict scip-clang documents can override that guess, but generic and
static lanes cannot.

Correct semantic model for headers

compile_commands.json contains translation units, not header rows. A header's
semantic language and preprocessor state come from each including translation
unit. Therefore:

Inclusion context	Required graph view
only C TUs	C
only C++ TUs	C++
both, or different defines/targets	separate semantic views in one atomic universe

Do not replace the extension guess with “look up the header in the compilation
database”; no direct entry exists. Build an include-graph/TU ownership map from
the actual index. A shared header may have several legitimate expansions.

Why SCIP plus a second AST pass is rejected as the final route

scip-clang is a maintained portable fallback, but:

• it builds its own descriptor scheme rather than publishing Clang USRs;
• internal-linkage twins and macros have identity risks;
• it deduplicates well-behaved headers through an arbitrary including TU;
• the common artifact does not prove call/read/write/construction roles; and
• it is batch-only.

A separate libclang/AST enrichment pass would parse every TU a second time,
roughly paying another compiler analysis and losing clangd's resident preambles
and invalidation. That violates the speed objective.

Clang already exposes the required vocabulary. Its
SymbolRole includes
declaration, definition, reference, read, write, call, dynamic, implicit,
address-of and relation roles. IndexDataConsumer receives semantic
occurrences. Clangd already stores content-addressed per-file background-index
shards and incremental updates through
BackgroundIndex
and
BackgroundIndexStorage.
The current clangd shard/collector drops facts the graph needs.

Decision: thin clangd fork

Maintain a narrowly scoped, commit-pinned clangd fork that changes collection,
shard schema and bulk export:

1. retain every relevant Clang role bit and relation endpoint;
2. retain local/file-static declarations and their enclosing semantic owner;
3. store declarations and definitions with full spans;
4. store includes/modules, macros and expansion/spelling evidence;
5. record the exact compile-command digest and TU/configuration identity;
6. version the on-disk shard schema; and
7. add a samchon/graphSnapshot request/CLI that freezes the required shard
   set, streams it through [Bulk index] Replace symbol-by-symbol LSP scans with compiler-owned snapshots #63's protocol, and refuses to commit if the
   compilation database or relevant shard generation moves.

This is more than adding transport, so it supersedes the earlier “do not fork
clangd merely to add transport” recommendation. An upstreamable design is still
preferred, but the semantic loss in current storage justifies the thin fork.

Fact mapping

• contains: semantic owner relation, including locals and anonymous
  declarations;
• imports: include/import/module evidence, keeping textual include distinct
  in provenance;
• calls: call role and called relation;
• accesses: read/write roles;
• instantiates: constructor calls and construction expressions;
• type_ref: type-role references;
• extends/implements: direct base relation, with C++ class/protocol modeling
  documented rather than guessed;
• overrides: direct override relation;
• references: reference role;
• exports: linkage/module export only where the language/build proves it;
• decorates: attributes from the same AST/index event if retained;
• tests and renders: framework enrichers, not core Clang facts.

A virtual call normally proves a declaration plus the override closure, not one
executed implementation. Store candidates in structured unresolved data.
Emit exact dispatches only for a compiler-proven closed/final target.
Inactive preprocessor branches are conditional-build.

Identity and universe

Use Clang USRs as the semantic core, then add the coordinates required where a
USR is not globally sufficient:

• compile target/configuration;
• TU/file discriminator for internal linkage;
• module/target triple and language mode;
• macro spelling/definition identity plus expansion evidence; and
• source owner for local entities.

Exact fixtures must prove:

• same-named static functions in two C files do not collide;
• anonymous namespaces and anonymous types remain distinct;
• overload insertion/reordering does not rename unrelated functions;
• templates, specializations and instantiations are not conflated;
• a macro survives unrelated line insertion where the compiler identity does;
  and
• one .h included as C and C++ retains both views without cross-contamination.

The universe includes every compilation command, driver, working directory,
defines, standard, include/module paths, sysroot, target, generated header and
dependency digest. A compilation database is necessary but not sufficient when
wrappers or generated inputs are unresolved. Fail closed to clangd/static
fallback when preparation cannot prove them.

Incremental and platform design

Reuse clangd's preambles, AST caches and digest-based background index.

• no-op: validate/load the same explicit shard set;
• source edit: re-export clangd's changed source shard and affected header/TU
  views;
• header change: invalidate every including TU reported by the include graph;
• compilation database/config/generated-header change: invalidate the affected
  command universe;
• delete/rename: explicit shard deletion; and
• mid-export shard movement/cancel/crash: keep the prior graph.

Do not run clangd-indexer and call it incremental; its monolithic export is
batch. Do not force --jobs=1 or producer-side deterministic scheduling:
master already canonicalizes incoming facts, and serializing scip-clang caused
orders-of-magnitude slowdowns without fixing header selection.

scip-clang's release assets do not provide a stock Windows binary. The final
fork must have reproducible Windows, Linux and macOS builds or explicitly
decline. Compiler-wrapper and MSVC-clang-cl compatibility belong in the
platform matrix.

Implementation map

1. Fix source discovery so .h reaches ownership resolution before language
   partitioning; add generic/static regression tests.
2. Prototype full-role collection in a pinned clangd fork and publish a schema
   capability handshake.
3. Add a provider beside the existing scip-clang entry; keep the latter as
   portable fallback.
4. Adapt changed shards through the common transaction and preserve dual C/C++
   language slices.
5. Add exact C and C++ fixtures before real Redis/LevelDB smoke.
6. Package and verify the fork on all supported platforms.

Acceptance

• No per-symbol LSP requests and no duplicate TU semantic analysis in the
  final path.
• Headers are owned by include/TU configuration, never a global extension
  guess.
• Exact fixtures cover internal linkage, macros, includes, modules,
  generated headers, overloads, templates/specializations, constructors,
  fields, inheritance, overrides and virtual calls.
• All 15 fact families carry explicit coverage/unresolved status.
• Same header in multiple TUs/configurations is neither arbitrarily
  deduplicated nor duplicated as the same identity.
• No-op, edit, header, compdb, generated input, create/delete/rename,
  compiler error, cancel, crash and retry are atomic.
• Cold time is compared with native clangd background indexing; no-op p95
  targets below 250 ms and one-TU body edit is proportional to the affected
  shard closure.
• Windows, Linux and macOS selection/packaging/fallback are proven.
• Redis and LevelDB smokes assert exact representative calls, access roles,
  type and inheritance facts, not counts.
• Focused tests, C/C++ experiments, pnpm build, pnpm test, and
  pnpm coverage pass.

Primary references

• clangd indexing design
• Clang indexing roles
• IndexDataConsumer
• clangd BackgroundIndex
• scip-clang

[samchon]

Provider recon at graph 71e35af. The .h question resolves against this issue's framing, and scip-clang's identity has a gap that may reproduce the exact defect #65 exists to fix.

The .h question: compilation databases contain no headers

This issue says .h is classified as C from its extension even when the authoritative compile command parses it as C++. That is correct about the bug — languageOf.ts:7 is an unconditional if (ext === ".h") return "c"; and it should go. But the fix is not "ask the compilation database".

Compilation databases contain translation units only. Headers are never entries. A compile command therefore cannot say what language a .h is — header language is inherited from the including TU.

• clangd guesses: "if no command is available for a header but a file has been opened that transitively includes it, that file's command will be used," selected by filesystem-path heuristics. Known defect: clangd#519, "Source file chosen to infer compile commands for a header sometimes has the wrong language".
• scip-clang sidesteps the guess by indexing headers through their including TU, and dedupes: "For well-behaved headers... it is sufficient to emit a SCIP Document only once, from any TU that includes the header."

The honest model is include-graph ownership:

Case	Truth
.h included only by C TUs	C
.h included only by C++ TUs	C++ (today: silently mislabeled C)
.h included by both	legitimately both, with two different expansions

The third row is where the C/C++ slice split actually breaks. Since the campaign keeps C and C++ as separate fixture/acceptance rows off one shared provider, a dual-included header must map to two identities under one preparation fence (step 6 here, step 4 in #67). scip-clang's "emit once from any TU" dedup picks one arbitrarily — it is optimizing away precisely the distinction the acceptance rows demand.

Requested amendment: replace the extension guess with include-graph ownership from TUs, not a compdb lookup; and expect to need --package-map discipline or a per-TU shard read rather than the merged index.

Identity: the highest-value thing to confirm before committing

scip-clang does not use Clang USRs. It builds its own descriptor scheme (cxx . <package> $ <descriptors>) in indexer/SymbolFormatter.cc.

The good news, and it is genuinely good: the overload/template disambiguator is a hex hash of the canonical type string (getFunctionDisambiguator()). Signature-derived, so it survives reordering and insertion — materially stronger than scip-java, scip-dotnet, and scip-kotlinc, which are all ordinal (see the #65 comment).

The bad news:

• Internal linkage appears to have no discriminator. getContextSymbol() routes TranslationUnitDecl/ExternCContextDecl to getLocationBasedSymbolPrefix(), which returns SymbolBuilder{packageId, /*descriptors*/ {}} — package ID only, no file identity. Anonymous namespaces get $anonymous_namespace_{filepath} and anonymous types get $anonymous_type_{hash}_{counter}, but file-scope static functions/variables appear not to. If so, static void foo() in a.c and static void foo() in b.c collide on one symbol string.

  This is read from SymbolFormatter.cc through a summarizing fetch, not executed. It is the single highest-value thing to confirm with a fixture: two .c files, same-named static function, check for one symbol or two. Internal linkage is an acceptance row here and [Bulk index] Introduce overload-safe provider-native symbol identities #65 lists "C/C++ internal linkage and translation units" as a known collider — if confirmed, scip-clang reproduces the exact defect [Bulk index] Introduce overload-safe provider-native symbol identities #65 exists to fix, and identity v2 must add a TU/file discriminator on top of the provider-native key rather than trusting it.

• Macros are keyed filepath:line:column → position-dependent. A line insertion above a macro changes its identity, violating [Bulk index] Introduce overload-safe provider-native symbol identities #65's "a line insertion or body-only edit keeps every provider-stable ID unchanged".

• Locals are local {counter}, sequential per document → document-scoped, must be path-prefixed.

• Templates: the announcement states "Template specializations require future work" — a known upstream gap against this issue's template acceptance row.

Invocation, and the platform blocker

scip-clang v0.4.0 (2026-02-23), built on Clang 21.

scip-clang --compdb-path=path/to/compile_commands.json

"You must invoke scip-clang from the project root, not from a subdirectory." — README

Release assets: scip-clang-x86_64-linux (glibc ≥2.16), scip-clang-arm64-darwin, scip-clang-dev-x86_64-linux. No Windows binary. No arm64-linux. No x86_64-darwin.

This issue requires "Windows and POSIX preparation paths are proven". Today there is no stock Windows path — a source build (untested, as this issue notes) or a Linux container is required. Recorded as a known gap; the provider code is platform-neutral and the real-toolchain proof belongs to CI.

Resources: ~2 MB temp/TU, 2 MB /dev/shm per core on Linux, ~2 GB RAM/core. Indexing costs ~30–50% over a parallel type-check baseline, ~50% slower than lsif-clang.

Build prep: CMake -DCMAKE_EXPORT_COMPILE_COMMANDS=ON; Bazel via hedronvision/bazel-compile-commands-extractor; Meson (ninja backend); Make via Bear. The existing ensureCompileCommands.ts already does the CMake path best-effort into a temp dir and is reusable, though it only checks root/compile_commands.json and root/build/compile_commands.json.

Edge kinds

Proven: contains (descriptor nesting + enclosing_range), references (occurrences).

Conflated: extends/implements arrive as one is_implementation flag (documented example: Dog::sound() → Animal::sound()), and C++ has no interface/class distinction, so the split is our modeling choice over one undifferentiated flag — needs direction tests. overrides uses the same flag, separable only by symbol kind.

Not provable: calls, instantiates (a constructor call and a type reference are indistinguishable in the role bitset), type_ref as such, dispatches, decorates (C++ attributes are absent from SCIP), tests (the Test role exists but no C++ framework is modeled). SCIP has no call role at all — see the #69 comment. exports is N/A pre-C++20; linkage is not export.

Partial: accesses (ReadAccess/WriteAccess exist in the schema; verify scip-clang actually sets them per occurrence — do not assume), imports (scip-clang navigates #include pragmas and the Import role exists, but #include is textual inclusion, not module import — the mapping is a semantic decision).

Required sidecar: a Clang AST/libclang pass keyed on the same compile_commands.json generation, emitting call-expr → callee, CXXConstructExpr → type, member-expr read/write, and attribute facts, joined to scip-clang symbols by descriptor. That is a second Clang parse of every TU — roughly doubling the ~30–50% overhead. Worth stating in the issue, since it is the most expensive enrichment in the campaign.

Incremental

Cold-only, as this issue already states. No daemon, no incremental mode, no documented cache invalidation; Design.md only offers "Workers may optionally cache data across TUs for performance." Milestone 1 = full rebuild.

clangd's background index (.cache/clangd/index/, per-file .idx, RIFF) is the only incremental substrate, and it is clangd-loadable, not interchange. clangd-indexer --executor=all-TUs is explicitly batch and its RelationKind enum is only BaseOf and OverriddenBy — strictly weaker than SCIP. This issue's "do not label clangd-indexer's export incremental" is correct.

Provenance

Tool version, Clang 21, compile_commands.json as the build-config fingerprint, diagnostics (SCIP Diagnostic), and Document.text for source fingerprints are all available. Better than most of the campaign.

Full recon: .wiki/bulk-index-providers/recon-c-cpp-java.md.

[samchon]

Campaign evidence — the extension guess currently drops a C++ project's headers entirely

While sealing #69's trust boundaries in #139 I confirmed this issue's fourth Evidence bullet against the shipped registry and a real CI artifact. It is worse than a misclassification: for a header-first C++ project the headers are never walked at all.

LANGUAGE_SPECS.ts gives c the extensions [".c", ".h"] and cpp the extensions [".cc", ".cpp", ".cxx", ".c++", ".hh", ".hpp", ".hxx", ".h++"]. .h appears only under c, and languageOf.ts opens with an unconditional if (ext === ".h") return "c"; before consulting the specs at all.

Two consequences follow, and they are different failures:

1. selectGraphSources walks allExtensions(requested). A build that requests cpp therefore never enumerates .h at all — not misfiled, absent. The pinned cpp experiment corpus is fmtlib/fmt, whose public API lives in include/fmt/*.h; the lane's 6,732 nodes and 87,185 edges come from its .cc translation units while the headers that declare almost everything a consumer would ask about are outside the walk.
2. A build that requests both c and cpp does enumerate them, but partitions every .h into the c slice, so a C++ header is handed to the C lane's expectations regardless of the translation units that actually include it.

The strict lane is already immune, which localises the fix. adaptScipIndex resolves a document's language as languageFromScip(document.language) ?? props.languageOf(file), so a SCIP producer that states its own per-document language overrides the guess. The defect belongs to discovery and to the generic and static lanes, which is where this issue's approach step 2 places it.

That makes header ownership separable from the scip-clang work in this issue. Ownership has to come from compile-command or translation-unit context, and where no compilation database exists the project's own evidence — which of C and C++ its translation units are — is still a better authority than a global constant. Whatever the resolution, allExtensions has to stop being the gate that decides a header is unreachable before ownership is ever consulted, or the acceptance case "a .h compiled as C++, C, and both under distinct configurations" cannot be expressed.

Not fixed in #139: that pull request sealed the SCIP enrichment and publication boundaries and did not take on the C/C++ provider. Recording the evidence here so the fix starts from a measured failure rather than the original inference.