refactor(exaforce): isolate schema pruning as runtime monkeypatch#8
Merged
Conversation
…tten. This is to improve llm request/response time and reduce timeout frequency.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Pruning moves to the fork-only exaforce runtime patch (subsequent tasks).
Runtime now matches PR #8. The two reverted upstream test files assert the un-pruned schema and fail by design (see docs/superpowers/EXPECTED_TEST_FAILURES.md).
These were process/design docs, not product deliverables. The functional EXPECTED_TEST_FAILURES.md is kept (referenced by the PR description).
smoy
approved these changes
Jul 10, 2026
smoy
left a comment
There was a problem hiding this comment.
you may still want to follow up to figure out how we should have a sanity CI test suite, just so we know we didn't accidentally break something.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
NVIDIA/SkillSpectormerge cleanly.src/skillspector/exaforce/package applied onimport skillspector.import skillspectorfails loudly withPatchDriftErrorinstead of silently going stale.Details
Why a monkeypatch instead of direct edits. Editing upstream files (
llm_analyzer_base.py,meta_analyzer.py, the two semantic analyzers) produces a merge conflict on every upstream sync. Moving the change into the fork-onlysrc/skillspector/exaforce/package keeps those files byte-identical to upstream — the net diff vsmaintouches no upstream source/test file except a 4-line activation block insrc/skillspector/__init__.py.Mechanism.
exaforce.apply_patches()mutates the Pydantic models in place —del model.model_fields[...]thenmodel_rebuild(force=True)on both the leaf model and its container (the container caches the nested schema) — and trims three prompt module-globals via a guardedstr.replace(). Mutating in place rather than swapping classes preserves class identity, soresponse_schemabindings andisinstancechecks keep working untouched.LLMFinding(_schema_patches.py) — dropsexplanation,remediation, and stops forwarding them into_findingMetaAnalyzerFinding— dropsintent,impactMetaAnalyzerResult— dropsoverall_assessmentand its_parse_stringified_assessmentvalidatorANALYZER_PROMPTs + the metaPER_FILE_ANALYSIS_PROMPT(_prompt_patches.py) — trim the now-unused line-number / intent / impact instructionsActivation. A two-line block at the end of
src/skillspector/__init__.pycallsapply_patches()afterfrom skillspector.graph import ...(so every target module is already loaded — no circular import). Pruning is therefore the default on every entry path: CLI, MCP server, and the benchmark's library import.Why two upstream test files fail — by design. Because the upstream test files are also kept at parity,
tests/nodes/test_llm_analyzer_base.pyandtests/nodes/test_semantic_quality_policy.pyassert the un-pruned schema and so fail at runtime (4 tests). This is a deliberate trade: full upstream parity was chosen over a green upstream suite, to avoid the ongoing burden of keeping forked test assertions in sync with upstream. The exact expected-failure set is recorded indocs/superpowers/EXPECTED_TEST_FAILURES.mdas a regression tripwire — a failure there is only a real problem if it is not one of those four pruned-key assertions (e.g. an import/collection error). The fork-onlytests/exaforce/suite covers the patch behavior itself.Benchmark. Runtime behavior is unchanged from the original direct-edit version of this PR, so the earlier main-vs-PR comparison still holds: report.md
Tests: fork-only
tests/exaforce/suite 10/10; full suite 4 failed (all documented inEXPECTED_TEST_FAILURES.md) / 1261 passed.Implementation Plan
Reproduce this PR's runtime effect from an isolated
exaforcepackage while restoring every upstream file to parity.src/skillspector/exaforce/_patchlib.pyPatchDriftErroron drift)src/skillspector/exaforce/_schema_patches.pymodel_rebuildsrc/skillspector/exaforce/_prompt_patches.pysrc/skillspector/exaforce/__init__.pyapply_patches()(_PATCHEDflag)src/skillspector/__init__.pyapply_patches()call on importtests/exaforce/**docs/superpowers/EXPECTED_TEST_FAILURES.mdVerification:
git diff <upstream> -- <the 6 files>is empty (parity); thetests/exaforce/suite is green; the 4 full-suite failures matchEXPECTED_TEST_FAILURES.mdand are bounded to the two reverted files.