Skip to content

refactor(exaforce): isolate schema pruning as runtime monkeypatch#8

Merged
will-exaforce merged 11 commits into
mainfrom
prune-unused-keys-in-prompt-schemas
Jul 11, 2026
Merged

refactor(exaforce): isolate schema pruning as runtime monkeypatch#8
will-exaforce merged 11 commits into
mainfrom
prune-unused-keys-in-prompt-schemas

Conversation

@will-exaforce

@will-exaforce will-exaforce commented Jul 2, 2026

Copy link
Copy Markdown

Summary

  • Reproduces this PR's original LLM schema/prompt pruning (fewer keys → smaller structured-output requests → fewer LLM timeouts) without editing any upstream-tracked source file, so future syncs from NVIDIA/SkillSpector merge cleanly.
  • The four source files this PR originally edited are reverted to upstream parity; the pruning now happens at runtime from a fork-only src/skillspector/exaforce/ package applied on import skillspector.
  • Every mutation is guarded — if an upstream rename/rewrite removes a target field or prompt sentence, import skillspector fails loudly with PatchDriftError instead of silently going stale.
  • Net runtime behavior is identical to the original direct-edit version of this PR (same pruned schema, same trimmed prompts), so the attached benchmark comparison still applies.

Details

Why a monkeypatch instead of direct edits. Editing upstream files (llm_analyzer_base.py, meta_analyzer.py, the two semantic analyzers) produces a merge conflict on every upstream sync. Moving the change into the fork-only src/skillspector/exaforce/ package keeps those files byte-identical to upstream — the net diff vs main touches no upstream source/test file except a 4-line activation block in src/skillspector/__init__.py.

Mechanism. exaforce.apply_patches() mutates the Pydantic models in place — del model.model_fields[...] then model_rebuild(force=True) on both the leaf model and its container (the container caches the nested schema) — and trims three prompt module-globals via a guarded str.replace(). Mutating in place rather than swapping classes preserves class identity, so response_schema bindings and isinstance checks keep working untouched.

  • LLMFinding (_schema_patches.py) — drops explanation, remediation, and stops forwarding them in to_finding
  • MetaAnalyzerFinding — drops intent, impact
  • MetaAnalyzerResult — drops overall_assessment and its _parse_stringified_assessment validator
  • Both semantic ANALYZER_PROMPTs + the meta PER_FILE_ANALYSIS_PROMPT (_prompt_patches.py) — trim the now-unused line-number / intent / impact instructions

Activation. A two-line block at the end of src/skillspector/__init__.py calls apply_patches() after from skillspector.graph import ... (so every target module is already loaded — no circular import). Pruning is therefore the default on every entry path: CLI, MCP server, and the benchmark's library import.

Why two upstream test files fail — by design. Because the upstream test files are also kept at parity, tests/nodes/test_llm_analyzer_base.py and tests/nodes/test_semantic_quality_policy.py assert the un-pruned schema and so fail at runtime (4 tests). This is a deliberate trade: full upstream parity was chosen over a green upstream suite, to avoid the ongoing burden of keeping forked test assertions in sync with upstream. The exact expected-failure set is recorded in docs/superpowers/EXPECTED_TEST_FAILURES.md as a regression tripwire — a failure there is only a real problem if it is not one of those four pruned-key assertions (e.g. an import/collection error). The fork-only tests/exaforce/ suite covers the patch behavior itself.

Benchmark. Runtime behavior is unchanged from the original direct-edit version of this PR, so the earlier main-vs-PR comparison still holds: report.md

Tests: fork-only tests/exaforce/ suite 10/10; full suite 4 failed (all documented in EXPECTED_TEST_FAILURES.md) / 1261 passed.

Implementation Plan

Reproduce this PR's runtime effect from an isolated exaforce package while restoring every upstream file to parity.

# Layer File Change
1 Revert 4 source + 2 test files Restore to pre-PR upstream parity
2 Patch lib src/skillspector/exaforce/_patchlib.py Guarded primitives (raise PatchDriftError on drift)
3 Schema patch src/skillspector/exaforce/_schema_patches.py Prune model fields + model_rebuild
4 Prompt patch src/skillspector/exaforce/_prompt_patches.py Trim prompt module-globals
5 Orchestrator src/skillspector/exaforce/__init__.py Idempotent apply_patches() (_PATCHED flag)
6 Activation src/skillspector/__init__.py 2-line apply_patches() call on import
7 Tests tests/exaforce/** Guard-primitive unit tests + subprocess patch/activation tests
8 Docs docs/superpowers/EXPECTED_TEST_FAILURES.md Documented expected-failure set

Verification: git diff <upstream> -- <the 6 files> is empty (parity); the tests/exaforce/ suite is green; the 4 full-suite failures match EXPECTED_TEST_FAILURES.md and are bounded to the two reverted files.

…tten. This is to improve llm request/response time and reduce timeout frequency.
@will-exaforce will-exaforce requested review from pupapaik and smoy July 2, 2026 19:54
will-exaforce and others added 9 commits July 10, 2026 07:37
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Pruning moves to the fork-only exaforce runtime patch (subsequent tasks).
Runtime now matches PR #8. The two reverted upstream test files assert the
un-pruned schema and fail by design (see docs/superpowers/EXPECTED_TEST_FAILURES.md).
@will-exaforce will-exaforce changed the title Remove keys in schema prompt not being used or actively being overwritten. refactor(exaforce): isolate schema pruning as runtime monkeypatch Jul 10, 2026
These were process/design docs, not product deliverables. The functional
EXPECTED_TEST_FAILURES.md is kept (referenced by the PR description).

@smoy smoy left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you may still want to follow up to figure out how we should have a sanity CI test suite, just so we know we didn't accidentally break something.

@will-exaforce will-exaforce merged commit 7f81e10 into main Jul 11, 2026
0 of 4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants