Skip to content

Commit 801d952

Browse files
hotlongclaude
andauthored
fix(scripts): derive the skill-frame-freshness self-test fixture from the live documents (#8109)
`pnpm check:skill-frame-freshness` was red on a clean `main`, and it was red in the worst possible place: `package.json` runs `--self-test && <the scan >`, so a fixture failure short-circuits the real check. The gate stayed in the gate list, reading as coverage, while scanning nothing — every PR touching the guarded skill files sailed past it. The fixture, not the gate, was broken. It was a table of literal find/replace rules that manufactured the historical two-axis specimen by re-spelling the real documents' prose; the 2026-08-12 principles-only rewrites reworded one axis line, the rule matched nothing, and the fixture threw. The skill files were correct throughout, and the sibling `check:skill-frame-sync` — which parses rather than re-spells — survived the same rewrite untouched. So the specimen is now derived the way the gates already read these documents: count sentences through each copy's own `start`/`binding` anchor, the axis entry to drop through the shared entry parser plus `AXIS_MAP`, and in-file count mentions through a shared mention scan. Nothing here spells a sentence of the frame, so any rewording, re-indentation or reflow carries the fixture with it. This does not make the gate tautological: what is derived is how the SPECIMEN is manufactured, never what the gate compares. The self-test still commits the demoted specimen and the real documents as two commits in a temp repo and demands the gate call the older one behind — a demotion that silently did nothing fails loudly in `verifyDemoted()`, which names the file and line to edit. `check-skill-frame-sync.mjs` gains three exports for this (`ENTRY_START`, `axisEntryStarts`, `frameCountMentions`) and its mention check now runs through the last of them, so the two scripts cannot disagree about what a mention is. Cases 8 and 9, which broke a declaring sentence by spelling it out, are derived from the same anchors for the same reason. Co-authored-by: Claude <noreply@anthropic.com>
1 parent f5434b0 commit 801d952

2 files changed

Lines changed: 338 additions & 85 deletions

File tree

scripts/check-skill-frame-freshness.mjs

Lines changed: 278 additions & 71 deletions
Original file line numberDiff line numberDiff line change
@@ -109,7 +109,15 @@ import { tmpdir } from 'node:os';
109109
import { dirname, join, resolve } from 'node:path';
110110
import { fileURLToPath } from 'node:url';
111111

112-
import { AXIS_MAP, COPIES, analyzeCopy, runAllChecks } from './check-skill-frame-sync.mjs';
112+
import {
113+
AXIS_MAP,
114+
COPIES,
115+
ENTRY_START,
116+
analyzeCopy,
117+
axisEntryStarts,
118+
frameCountMentions,
119+
runAllChecks,
120+
} from './check-skill-frame-sync.mjs';
113121

114122
const REPO_ROOT = resolve(dirname(fileURLToPath(import.meta.url)), '..');
115123

@@ -467,73 +475,286 @@ function render(verdict, { root = REPO_ROOT } = {}) {
467475
// self-test — real temp git repositories, never the network
468476
// ---------------------------------------------------------------------------
469477

478+
/** The real documents, as committed on this branch — fixtures are never synthetic. */
479+
function realFrameFiles() {
480+
const files = new Map();
481+
for (const file of FRAME_FILES) files.set(file, readFileSync(join(REPO_ROOT, file), 'utf8'));
482+
return files;
483+
}
484+
485+
// ---------------------------------------------------------------------------
486+
// the two-axis specimen — DERIVED from the real documents, never re-spelled
487+
// ---------------------------------------------------------------------------
488+
//
489+
// The specimen turns the REAL three-axis documents into a COHERENT two-axis tree
490+
// — the #5866 shape. Coherent matters: every count sentence and every in-file
491+
// mention moves with the axes, which is precisely why the SYNC gate stays green
492+
// on it. The self-test asserts that green, so the derivation below is validated
493+
// by the sync gate itself rather than by its own say-so.
494+
//
495+
// DERIVED, NOT RE-SPELLED (#8024)
496+
// -------------------------------
497+
// The first version of this fixture was a table of literal find/replace rules —
498+
// a second hand-kept copy of the very prose these gates exist to police, and it
499+
// failed the way hand-kept copies do. On 2026-08-12 a principles-only rewrite
500+
// reworded one axis line; the rule spelling the old wording matched nothing, the
501+
// fixture threw, and because package.json runs `--self-test && <the scan >` the
502+
// gate died BEFORE scanning anything. It stayed in the gate list — reading as
503+
// coverage while protecting nothing — and the skill files it guards were correct
504+
// the whole time. A guard that cannot run is worse than an absent one.
505+
//
506+
// So the mutation is now located the way the gates themselves read these
507+
// documents:
508+
// • the two count sentences — through each copy's own `start`/`binding` anchor;
509+
// • the axis entry to drop — through `axisEntryStarts()` + `AXIS_MAP`;
510+
// • the in-file count mentions — through `frameCountMentions()`.
511+
// Nothing here spells a sentence of the frame, so a rewrite that keeps the
512+
// frame's STRUCTURE — any wording, any indentation, any reflow — carries the
513+
// fixture with it. A rewrite that changes the structure still fails, loudly and
514+
// on purpose, in `verifyDemoted()`, which names the file and line to look at.
515+
//
516+
// What that leaves untouched is narrative prose stating a count that NEITHER gate
517+
// reads ("analyzed on **all three** axes" in a lead-in paragraph, "三轴冲突时").
518+
// The old table re-spelled those too. Deriving them is not possible without
519+
// hand-copying prose again, and demoting every "three" in a watched file would
520+
// hit sentences about unrelated axes — so they stay, and the specimen is
521+
// deliberately a structurally faithful two-axis tree rather than a
522+
// copy-edited one. Nothing in either gate's criterion reads them.
523+
//
524+
// WHY THIS IS NOT THE TAUTOLOGY IT LOOKS LIKE
525+
// -------------------------------------------
526+
// "Derive the fixture from the live files" is vacuous when it makes a check
527+
// compare the files to themselves. It does not here, and the difference is worth
528+
// stating because the reasonable objection lands on the wrong half of the script:
529+
// what is derived is how the SPECIMEN IS MANUFACTURED, not what the gate compares.
530+
// The gate's own comparison (this tree's structure vs `origin/main`'s) never
531+
// touches any of this. The self-test still commits the demoted specimen and the
532+
// real documents as two different commits in a temp repo and demands the gate
533+
// call the older one BEHIND — a claim the derivation cannot fake: a demotion that
534+
// silently did nothing would make specimen and real identical, and case 1 would
535+
// fail with "expected error, got ok" instead of quietly passing.
536+
537+
/**
538+
* The axis the specimen drops. Named by AXIS_MAP id, never by its prose: it is
539+
* the FIRST axis, which is the #5130 damage this whole gate family exists to
540+
* prevent (a dev agent handed a frame with no business-need axis cannot tell).
541+
*/
542+
const DROPPED_AXIS = 'business-need';
543+
544+
/**
545+
* How each copy's language spells a count. This is a numeral vocabulary, not a
546+
* copy of the frame's prose — the one thing a mutation that removes an axis has
547+
* to write, and nothing a rewrite of the escalation section can strand. The
548+
* spellings are read back through the sync gate's own parser by
549+
* `verifyDemoted()`, which requires the demoted copy to parse as N-1 axes.
550+
*/
551+
const COUNT_WORDS = {
552+
en: ['zero', 'one', 'two', 'three', 'four', 'five', 'six'],
553+
zh: ['零', '一', '两', '三', '四', '五', '六'],
554+
};
555+
556+
const lineOf = (text, index) => text.slice(0, index).split('\n').length;
557+
558+
function fixtureError(where, message) {
559+
return new Error(
560+
`self-test fixture could not be derived from ${where}: ${message}\n` +
561+
` The specimen is derived from the live documents (#8024), so this is a change of ` +
562+
`STRUCTURE, not of wording — reword the frame freely, but adding, removing or ` +
563+
`reshaping an axis means editing the derivation in scripts/check-skill-frame-freshness.mjs.`,
564+
);
565+
}
566+
567+
/** Rewrite a captured numeral token (`three`, `三条`) to the count one lower. */
568+
function renumber(numeral, from, to) {
569+
for (const words of Object.values(COUNT_WORDS)) {
570+
if (words[from] && numeral.includes(words[from])) return numeral.replace(words[from], words[to]);
571+
}
572+
return null;
573+
}
574+
575+
/**
576+
* Un-mark an axis entry's first line so it stops being an entry. `ENTRY_START`
577+
* admits exactly two shapes and this inverts whichever one the line is: a bullet
578+
* loses its marker (the indent is preserved, so the prose still reads as part of
579+
* the paragraph above), and an `**Axis …**` head is indented, which is enough
580+
* because that alternative is anchored at column 0. The result is asserted
581+
* against the real `ENTRY_START`, so a third shape added to the parser tomorrow
582+
* fails here instead of silently producing a specimen that still has N axes.
583+
*/
584+
function unmarkEntryLine(line, where) {
585+
let out = line.replace(/^(\s*)[-*](\s)/, '$1 $2');
586+
if (out === line) out = ` ${line}`;
587+
if (ENTRY_START.test(out)) {
588+
throw fixtureError(where, `un-marking the axis entry left it still matching ENTRY_START: "${out.trim().slice(0, 80)}"`);
589+
}
590+
return out;
591+
}
592+
593+
/**
594+
* The edits that demote ONE copy by an axis. Offsets are into the real file text
595+
* and applied later in one pass, so every edit is located in the untouched
596+
* document and none of them can shift another out from under itself.
597+
*/
598+
function demotionEdits(copy, text, analysis) {
599+
const edits = [];
600+
const from = analysis.declared;
601+
const to = from - 1;
602+
const where = `${copy.file} (${copy.id})`;
603+
604+
// 1. the count sentences, found through the copy's own anchors.
605+
for (const [what, anchor] of [['declaring', copy.start], ['binding', copy.binding]]) {
606+
const hits = [...text.matchAll(new RegExp(anchor, 'g'))];
607+
if (hits.length !== 1) {
608+
throw fixtureError(where, `the ${what} anchor matched ${hits.length} time(s), expected exactly 1`);
609+
}
610+
const hit = hits[0];
611+
const renumbered = renumber(hit[1], from, to);
612+
if (renumbered == null) {
613+
throw fixtureError(
614+
`${where}:${lineOf(text, hit.index)}`,
615+
`the ${what} sentence writes its count as "${hit[1]}", which is not ${from} in any ` +
616+
`spelling COUNT_WORDS knows`,
617+
);
618+
}
619+
let rewritten = hit[0].replace(hit[1], renumbered);
620+
// English quantifies two as "both axes", never "all two axes" — the same
621+
// idiom the sync gate's NUMERALS table records.
622+
if (copy.lang === 'en' && to === 2) rewritten = rewritten.replace(/\ball\s+two\b/, 'both');
623+
edits.push({ start: hit.index, end: hit.index + hit[0].length, text: rewritten, priority: 0 });
624+
}
625+
626+
// 2. the axis entry itself, found through the shared entry parser.
627+
const section = text.slice(analysis.sectionStart, analysis.sectionEnd);
628+
const starts = axisEntryStarts(section);
629+
if (starts.length !== analysis.ids.length) {
630+
throw fixtureError(where, `${starts.length} entry line(s) for ${analysis.ids.length} parsed axes — the entry parser and this fixture disagree`);
631+
}
632+
const which = analysis.ids.indexOf(DROPPED_AXIS);
633+
if (which < 0) {
634+
throw fixtureError(where, `this copy has no \`${DROPPED_AXIS}\` axis to drop; it lists ${analysis.ids.join(' → ')}`);
635+
}
636+
const lineStart = analysis.sectionStart + starts[which];
637+
const nl = text.indexOf('\n', lineStart);
638+
const lineEnd = nl === -1 ? text.length : nl;
639+
edits.push({
640+
start: lineStart,
641+
end: lineEnd,
642+
text: unmarkEntryLine(text.slice(lineStart, lineEnd), `${where}:${lineOf(text, lineStart)}`),
643+
priority: 0,
644+
});
645+
646+
return edits;
647+
}
648+
470649
/**
471-
* The mutation set that turns the REAL three-axis documents into a COHERENT
472-
* two-axis tree — the #5866 specimen. Coherent matters: every count sentence and
473-
* every in-file mention moves with the axes, which is precisely why the sync gate
474-
* stays green on it. The self-test asserts that green, so this table is validated
475-
* by the sync gate itself: an incomplete mutation makes sync red and fails the
476-
* case loudly instead of quietly weakening it.
477-
*
478-
* `drop` un-marks the business-need entry so it stops being an axis entry (a
479-
* leading space is enough — the sync gate's ENTRY_START is anchored at column 0),
480-
* which is the #5130 damage this whole family exists to prevent.
650+
* Apply edits right-to-left. Anything overlapping an already-kept edit is
651+
* dropped by priority: a count mention that IS the declaring sentence (the zh
652+
* copy has exactly one) is already handled by the anchor edit, which rewrites
653+
* the same numeral.
481654
*/
482-
const TO_TWO_AXES = [
483-
// counts, bindings and mentions — global, so one rule covers several copies
484-
{ from: /on three fixed axes/g, to: 'on two fixed axes' },
485-
{ from: /on the three fixed axes below/g, to: 'on the two fixed axes below' },
486-
{ from: /on all three axes/g, to: 'on both axes' },
487-
{ from: /on \*\*all three\*\* axes/g, to: 'on **both** axes' },
488-
{ from: /three-axis(\s+)analysis/g, to: 'two-axis$1analysis' },
489-
{ from: /three-axis decision frame/g, to: 'two-axis decision frame' },
490-
{ from: /沿/g, to: '沿两条固定评估轴' },
491-
{ from: /;/g, to: '这两条轴给出理由;两轴冲突时' },
492-
// the axis entries themselves, one per copy
493-
{ from: '\n- **Real business need**: does this option serve', to: '\n **Real business need**: does this option serve' },
494-
{ from: '\n - **实际业务需求** — 每个方案先问', to: '\n **实际业务需求** — 每个方案先问' },
495-
{ from: '\n**Axis ① — real business need.**', to: '\n _Axis ① — real business need._' },
496-
{ from: '\n- Real business need — does the option serve', to: '\n Real business need — does the option serve' },
497-
];
498-
499-
function applyRules(text, rules) {
655+
function applyEdits(text, edits) {
656+
const kept = [];
657+
for (const e of [...edits].sort((a, b) => a.priority - b.priority || a.start - b.start)) {
658+
if (kept.some((k) => e.start < k.end && k.start < e.end)) continue;
659+
kept.push(e);
660+
}
500661
let out = text;
501-
const unapplied = [];
502-
for (const { from, to } of rules) {
503-
const next = out.replace(from, to);
504-
if (next === out) unapplied.push(String(from));
505-
out = next;
662+
for (const e of kept.sort((a, b) => b.start - a.start)) {
663+
out = out.slice(0, e.start) + e.text + out.slice(e.end);
506664
}
507-
return { text: out, unapplied };
665+
return out;
508666
}
509667

510-
/** The real documents, as committed on this branch — fixtures are never synthetic. */
511-
function realFrameFiles() {
512-
const files = new Map();
513-
for (const file of FRAME_FILES) files.set(file, readFileSync(join(REPO_ROOT, file), 'utf8'));
514-
return files;
668+
/**
669+
* The specimen must really BE the frame it claims to be — one axis fewer, the
670+
* survivors in the same order, still parseable by the gate that will judge it.
671+
* This is where a structural change to the frame is reported, and it is the
672+
* error a human is meant to act on.
673+
*/
674+
function verifyDemoted(files, expected) {
675+
for (const copy of COPIES) {
676+
const problems = [];
677+
const got = analyzeCopy({ ...copy, text: files.get(copy.file) }, AXIS_MAP, problems);
678+
const want = expected.get(copy.id);
679+
const reads = got ? `${got.declared} axes ${got.ids.join(' → ')}` : `unparseable:\n ${problems.join('\n ')}`;
680+
if (!got || got.declared !== want.length || got.ids.join(' → ') !== want.join(' → ')) {
681+
throw fixtureError(
682+
`${copy.file} (${copy.id})`,
683+
`the demoted copy reads ${reads}\n expected ${want.length} axes ${want.join(' → ')}`,
684+
);
685+
}
686+
}
515687
}
516688

517689
function twoAxisFrameFiles() {
518-
const files = new Map();
519-
const unapplied = [];
520-
for (const [file, text] of realFrameFiles()) {
521-
const r = applyRules(text, TO_TWO_AXES);
522-
files.set(file, r.text);
523-
unapplied.push(...r.unapplied.map((u) => `${file}: ${u}`));
524-
}
525-
// Each rule must land somewhere; a rule that matches nothing means the real
526-
// document drifted and the fixture is no longer the frame it claims to be.
527-
const everywhere = unapplied.filter((u) => {
528-
const rule = u.slice(u.indexOf(': ') + 2);
529-
return unapplied.filter((x) => x.endsWith(rule)).length === FRAME_FILES.length;
530-
});
531-
if (everywhere.length > 0) {
532-
throw new Error(`self-test fixture drifted — rule(s) matched nothing anywhere: ${[...new Set(everywhere)].join(', ')}`);
690+
const real = realFrameFiles();
691+
const edits = new Map([...real.keys()].map((f) => [f, []]));
692+
const expected = new Map();
693+
694+
for (const copy of COPIES) {
695+
const text = real.get(copy.file);
696+
const problems = [];
697+
const analysis = analyzeCopy({ ...copy, text }, AXIS_MAP, problems);
698+
if (!analysis) {
699+
throw fixtureError(
700+
`${copy.file} (${copy.id})`,
701+
`the REAL document does not parse, so there is nothing to demote — this is a ` +
702+
`\`pnpm check:skill-frame-sync\` failure first:\n ${problems.join('\n ')}`,
703+
);
704+
}
705+
edits.get(copy.file).push(...demotionEdits(copy, text, analysis));
706+
expected.set(copy.id, analysis.ids.filter((id) => id !== DROPPED_AXIS));
533707
}
708+
709+
// Mentions are a per-FILE property: "the three-axis analysis" elsewhere in a
710+
// watched file is what the sync gate's mention check reads, and a specimen
711+
// that left them at three would be an incoherent tree rather than an older one.
712+
for (const [file, text] of real) {
713+
const declared = COPIES.filter((c) => c.file === file).map((c) => expected.get(c.id).length + 1);
714+
for (const m of frameCountMentions(text)) {
715+
if (!declared.includes(m.count)) continue; // already disagrees with the frame — the sync gate's business, not ours
716+
const renumbered = renumber(m.numeral, m.count, m.count - 1);
717+
if (renumbered == null) {
718+
throw fixtureError(
719+
`${file}:${lineOf(text, m.index)}`,
720+
`a frame mention writes its count as "${m.numeral}", which is not ${m.count} in any ` +
721+
`spelling COUNT_WORDS knows`,
722+
);
723+
}
724+
edits.get(file).push({
725+
start: m.index,
726+
end: m.index + m.text.length,
727+
text: m.text.replace(m.numeral, renumbered),
728+
priority: 1,
729+
});
730+
}
731+
}
732+
733+
const files = new Map([...real].map(([file, text]) => [file, applyEdits(text, edits.get(file))]));
734+
verifyDemoted(files, expected);
534735
return files;
535736
}
536737

738+
/**
739+
* The real documents with ONE copy's declaring sentence deleted — a tree whose
740+
* frame cannot be read at all. Located through that copy's own anchor for the
741+
* same reason as everything above (#8024): a fixture that spelled the sentence
742+
* out would stop breaking anything the day the sentence is reworded, and cases
743+
* 8 and 9 would then be asserting on a document that parses perfectly.
744+
*/
745+
function withUnreadableCopy(real, copyId) {
746+
const copy = COPIES.find((c) => c.id === copyId);
747+
const text = real.get(copy.file);
748+
const hits = [...text.matchAll(new RegExp(copy.start, 'g'))];
749+
if (hits.length !== 1) {
750+
throw fixtureError(`${copy.file} (${copy.id})`, `the declaring anchor matched ${hits.length} time(s), expected exactly 1`);
751+
}
752+
return new Map([...real]).set(
753+
copy.file,
754+
text.slice(0, hits[0].index) + text.slice(hits[0].index + hits[0][0].length),
755+
);
756+
}
757+
537758
function writeFiles(dir, files) {
538759
for (const [file, text] of files) {
539760
const p = join(dir, file);
@@ -696,14 +917,7 @@ function selfTest() {
696917

697918
// --- 8: the frame moved on main, our anchors predate it ------------------
698919
{
699-
const moved = new Map([...real]);
700-
moved.set(
701-
'.claude/agents/os-dev.md',
702-
real.get('.claude/agents/os-dev.md').replace(
703-
'**Analyze every option on three fixed axes',
704-
'**Weigh every option on the standing axes',
705-
),
706-
);
920+
const moved = withUnreadableCopy(real, 'internal-dev');
707921
const { dir, b: current } = linear('anchors', real, moved);
708922
git(['checkout', '-q', git(['rev-parse', 'HEAD~1'], { cwd: dir }).stdout.trim()], { cwd: dir });
709923
setOriginMain(dir, current);
@@ -717,14 +931,7 @@ function selfTest() {
717931

718932
// --- 9: our own tree does not parse --------------------------------------
719933
{
720-
const broken = new Map([...real]);
721-
broken.set(
722-
'.claude/agents/os-dev.md',
723-
real.get('.claude/agents/os-dev.md').replace(
724-
'**Analyze every option on three fixed axes',
725-
'**Weigh every option sensibly',
726-
),
727-
);
934+
const broken = withUnreadableCopy(real, 'internal-dev');
728935
const { dir, b: current } = linear('broken-here', broken, real);
729936
git(['checkout', '-q', git(['rev-parse', 'HEAD~1'], { cwd: dir }).stdout.trim()], { cwd: dir });
730937
setOriginMain(dir, current);

0 commit comments

Comments
 (0)