production foundations: a gate that can fail, an exact environment, and an API that cannot quietly do the wrong thing - #4
Merged
Conversation
…profile
The licence flip is a change to which parser produces every number, and it was
about to be judged by a gate that could pass while none of the following held.
Each was reachable, and most are measured rather than hypothesised:
* `harness.evaluate()` returns {"error": ...} when the render fails and nothing
read the key. A renderer dying on all 16 documents scored zero failures.
* An absent metric hit `if v is None: continue`, so losing within2pt removed
the check rather than failing it -- the same dimension whose absence once hid
a 0.510 -> 0.291 regression.
* The baseline stored the NAMES of failing metrics. 04_exec_brief's live text
was on record as "known failing" at 0.941 and could have fallen to 0.10.
* page_match is a boolean, so a document already over could go from one page to
forty unremarked.
* Nothing compared the run to a manifest. Measured in a bare ubuntu:24.04
container: gen_corpus produced 3 of 16, printed "the corpus is incomplete,
numbers are NOT comparable", exited 0 -- and the gate scored those 3 against
the 16-document baseline and passed.
* REFINE=lanes returned only the refined lane's status, so the control lane
could regress freely.
* backend_parity exited on `regressions == 0` while ROADMAP and STATUS said two
documents were formally accepted. That disagreement was settled by marking
the CI step continue-on-error, which retired the one gate the whole
relicensing effort was aimed at.
What lands:
exactdoc/options.py One immutable PRODUCT profile. The API refined 0 times,
the CLI 2, and the published numbers came from a CI lane
that ran 3; a measurement describing no shipping
configuration is a coincidence. `backend` is a real
option now, so the instruments stop assigning over
convert.parse_pdf -- an injection that had become a no-op
setting an attribute nobody reads.
testkit/gate.py The decision, pure over measured results, so it can be
mutation-tested without a corpus or an oracle. Three
separate questions: regression (against the recorded
number), absolute (--absolute, and it fails today, on the
record), stale (a shortfall that now passes).
gate_baseline.json Schema 2: every gated metric of every document in both
lanes, numerically, plus the environment and the defect
ID each shortfall answers to. Recorded on Linux with
LibreOffice 24.2.7.2, and it reproduces the documented
numbers -- product 15/16 pages, within2pt 0.5118, dy50
0.62pt; raw 13/16, 0.3486, 2.20pt.
corpus_manifest.* The exact 16 documents, generator, dialect, page count.
No content hash: both generators embed timestamps, so a
hash would fail every run and be deleted within a week.
parity_policy.json The acceptance rule as data. 0 regressions, 11 same, 1
better, 2 expected divergences, 2 accepted shortfalls
bounded by recorded numeric floors. continue-on-error
comes off the step.
evidence.py One artifact every published number traces to. Its merge
skips None sections: the standalone step meant to fill in
the environment was passing the empty template's
`parity: None` over the verdict just recorded, and a green
run ended having forgotten its own parity result.
test_gate_mutations 30 tests, 106 assertions, about a second, no corpus
needed. One per false-green path above.
Two items closed by measuring instead of building:
* Superscript. parse_pdfium hardcodes False and the plan was to implement it.
backend_superscript.py measures the level that matters: the writer never sees
the parser's flag -- dialect and infer recover it from geometry -- and all 16
documents agree at the layout level. No code written.
* scripts/bootstrap.sh discovers Chromium and writes scripts/env.sh, which
nothing sources: every CI step is its own shell. CI only ever worked because
the GitHub runner image ships /usr/bin/google-chrome. _paths.py reads the
record itself, and the bare container now generates all 16.
Also: D10 in STATUS for the two rasterisation shortfalls the baseline needed an
ID for, attributed by measurement (figure-excluded coverage 0.988 / 0.978 /
0.966) and honest that rasterisation is the dominant cause on all three and the
whole cause on none. The dialect table summed to 17 for a 16-document corpus.
dy_p50 gets a proportional tolerance: it is the one gated metric that is not a
fraction, running 0.04pt to 101pt, and one absolute slack cannot serve both ends.
The full CI-equivalent sequence is verified green on the canonical Linux
environment. Phase 1 is unchanged and now stated honestly in STATUS section 3:
`fitz` is on the default runtime path well past the parser, so the "mechanical"
licence flip is not mechanical.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The roadmap called the licence flip mechanical: a dependency change and a default.
The built wheel disagreed. `fitz` was on the default runtime path in five stages
*past* the parser, so a wheel installed without PyMuPDF failed while importing the
writer, before any backend selection could happen.
docxout module scope `import fitz` -> gone
docxout table fitting MuPDF base-14 shaping -> Para.src_widths
docxout figures MuPDF pixmap of the clip -> Backend.render_clip
refine measurement get_text("dict") on both sides -> Backend.page_lines
verify rasterisation MuPDF pixmap samples -> Backend.render_page
ladder MuPDF base-14 shaping -> metrics.TextMetrics
tests/test_no_pymupdf.py is the proof, and it does not trust the code to avoid an
import: it installs a sys.meta_path finder that makes `fitz` *unimportable*, evicts
anything already loaded, and then converts a fixture per capability -- text-only,
tables, inline image, vector figure clip, multi-page with refinement, multi-column,
cover band, gdocs static profile -- and runs the closed loop through the permissive
path. That is stricter than a clean virtualenv, which cannot catch an import some
other module has already performed. All pass.
The writer's half cost nothing: both lanes re-measured, and not one of 224 values
moved (2 lanes x 16 documents x 7 gated metrics, compared exactly, not within
tolerance). Replacing base-14 shaping with the source's own recorded line widths is
not an approximation of the old answer -- for "is this column too narrow for
content that occupied one line in the source", it is a fact where there was a
prediction.
The refine half cost two documents, and the gate written last commit is what said
so: within-2pt 0.46 -> 0.31 on 03_tech_report_code and 0.60 -> 0.32 on
r1_reportlab_report. Cause is D2 in a second location. The loop measures
rendered_box_top - source_box_top over two differently-typeset documents; on the
core-14 source PDFium substitutes a generic ascent (0.905x size) for the real one
MuPDF reads from its base-14 table (1.075x), while on the Liberation render both
read the embedded metrics and agree. The subtraction keeps a ~0.17 x size bias,
about 1.7pt at 10pt, against a 2pt threshold.
Anchoring the loop on baselines cancels that perfectly and is measurably WRONG: the
incumbent's mean within-2pt goes 0.511 -> 0.478. It fixes 04_exec_brief
(0.22 -> 0.44) and breaks 05_memo (0.64 -> 0.48) and r1_reportlab_report
(0.60 -> 0.32) -- the same outcome as the line-box escalation already closed out in
D2, because the space_before chain the offsets feed is itself calibrated on box
tops. `refine.ANCHOR` keeps the switch and the measurement beside it so nobody
rediscovers this a third time.
So the two documents are ratified into parity_policy.json under D2, with floors,
joining the two already there. All four are core-14; every document that embeds its
fonts is untouched, which is the shape of the cause. **The accepted set grew from
two to four because the comparison got more honest, not because the backend got
worse**: until now refine.py read its measurement through PyMuPDF whichever backend
had parsed, so the candidate lane was pdfium parsing with MuPDF measuring -- a
configuration nobody could install. Parity is 0 regressions, 10 same, 2 expected
divergences, 4 accepted, and the end-to-end pdfium mean within-2pt is 0.4431 where
the hybrid reported 0.461. The honest number is the lower one.
Also here, both found by their own noise rather than by review:
* PDFium native handles were never closed, in `parse_pdfium.parse_pdf` and in
all three render paths. A parity run ended with pypdfium2 printing "The
following objects are still open and will now be closed" for 16 documents, 18
pages and 9 text pages. Interpreter exit collected them, which is not a
resource policy -- a worker converting a queue would hold a native document per
job until it died. Documents, pages and text pages now close in reverse order
of acquisition, and the warning is gone across 32 parses. Golden IR still 7/7,
so nothing was being read lazily off a handle that is now closed.
* Every LibreOffice invocation shared one profile -- a fixed temp path, for every
conversion in every process on the machine. Two concurrent conversions then
contend for it and one exits 0 with no output, which is the exact failure that
motivated using a dedicated profile at all. Default is now per-process.
* evidence.merge skipped nothing, so the final `evidence.py --out` step passed the
empty template's `parity: None` over the verdict just recorded, and a fully
green run ended with an artifact that had forgotten its own parity result. None
sections are now skipped, with a test.
* The writer's target mode was a module global that write_docx set and restored,
so two concurrent conversions with different targets could each observe the
other's line-height encoding. It travels in a frozen WriteCtx now.
* Changing the API default from 0 refine rounds to the product profile silently
gave edge_cases.py and exp_sweep.py three rounds and an oracle dependency they
did not want. Both now name RAW explicitly.
Superscript stays hardcoded False in the pdfium backend on purpose, and that is
measured rather than deferred: backend_superscript.py shows the writer never sees
the parser's flag.
What remains for the relicence is genuinely mechanical -- pypdfium2 as the runtime
dependency, pymupdf to `[mupdf]`, goldens re-frozen from pdfium, every gate number
re-recorded because the default parser changes -- plus one thing that is not an
engineering decision and is deliberately left alone here: a licensing review of the
Apache-2.0 distribution and the `[mupdf]` extra's wording.
Full CI-equivalent sequence green on the canonical Linux environment.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A review of the two preceding commits found ways the new gate could still pass
while meaning nothing. Each is closed here with a mutation test; the suite is now
150 assertions and still needs no corpus and no oracle.
**The parity gate now FAILS, and that is the point of the commit.** Judging every
dimension independently -- and comparing vertical drift, which was never compared
at all -- turned "0 regressions, 13 same, 1 better" into 2 unwaived regressions:
05_memo dy50 0.59 -> 1.89pt and f1_fpdf_brief dy50 0 -> 1.20pt. Neither is new
breakage. The comparison returned on the FIRST dimension outside its margin, so an
improvement suppressed every regression ordered after it, and dy_p50, doc_recall
and raster_frac were not in the list. Two documents were drifting by more than a
point while the gate said "same". Both are core-14, the same population as the
four D2 waivers, so they are plausible candidates for the same waiver -- they are
deliberately NOT waived. Five of sixteen documents excused under one defect is a
different product claim from two, and no measurement can authorise it.
Value hygiene. `None`, `True`, NaN, infinity and out-of-domain values all reached
the comparison operators and were scored. `isinstance(True, int)` is True and
`page_match` is a boolean living next to the metrics, so one misplaced key scored
1.0. NaN is worse: every comparison against it is False, so it passed its
threshold, its regression check and its stale check simultaneously, by failing to
be greater or less than anything. A `live_text_cov` of 1.7 cleared every threshold
in the file. Now: type, finiteness, semantic domain, page_match-vs-page-count
consistency, and renderer identity -- a lane scored against a different oracle
than the baseline is not comparable to it, and nothing checked.
Parity coverage. Comparing `set(reference)` with `set(candidate)` is satisfied by
two runs that dropped the SAME document, which is exactly what happened when a
conversion failed under both backends -- it vanished from both sides, the sets
matched, and it was never mentioned. Coverage is now anchored on the manifest, and
`run()` records structured `convert_error`/`eval_error` results instead of
printing and discarding them.
Waivers are bounded in both directions. `expected_divergence` excused a document
from every dimension, permanently: c5_graphics could have lost everything else it
had and still reported "expected-div". Divergences now carry the same numeric
floors and stale detection as accepted shortfalls -- and immediately show that
both are also worse on `raster_frac`, which nobody knew.
Backend precedence was inverted: `EXACTDOC_BACKEND` outranked an explicitly-passed
`ConversionOptions`, so an exported variable could redirect a caller that had named
its backend in code -- including the parity gate, whose whole job is to run one
named backend against another. Now explicit keyword > supplied options >
environment > PRODUCT, with a test per rung.
Recording a baseline is the most consequential write in the repository and had no
preconditions at all. `GATE_BASELINE=update` on a laptop, over a subset, with a
renderer failure in the middle, would overwrite the canonical record and every
later run would agree with it. Now refused off the canonical environment, refused
over an incomplete corpus, refused for a single lane, and written through a
temporary file plus `os.replace` so an interrupted write cannot truncate the
record. Same rules for the parity policy, and `--only` may no longer record floors.
Corpus identity was page count alone, which cannot see a document that was edited
without changing length. Added a content fingerprint over page geometry and
normalised text -- timestamp-independent, unlike the file bytes, which both
generators stamp. Recorded per extractor, because the two parsers genuinely
disagree (c2_paper2col and c4_i18n fingerprint differently, which is this
project's subject matter), and an extractor with no recorded fingerprint fails
rather than skips. Verified stable across a full corpus regeneration.
Resource and provisioning:
* `uv sync --frozen` -- gate.yml has claimed the lockfile was pinned truth in a
comment since before the flag was passed.
* PDFium bitmaps (`to_pil()` leaves the native buffer open) and web-link handles
(`FPDFLink_LoadWebLinks` returns a handle the caller owns) are now closed.
* Every LibreOffice invocation gets a fresh profile and removes it. It was one
fixed temp path shared by every process on the machine, so two concurrent
conversions reproduced the exact failure the profile existed to prevent.
`profile=` lets a batch caller keep one warm profile, which is what
testkit/harness.py deliberately wants.
Evidence artifacts now say what they do not contain: `git_state()` distinguishes
"clean tree" from "no repository here", `--stamp-git` stamps an artifact measured
in a container from the real checkout, and `validate()` reports an artifact as
NOT RELEASE-GRADE rather than letting a missing parity section read like a passing
one.
Docs corrected on three counts the review named: the explicit PDFium path is
permissive but the **shipped default still requires PyMuPDF** and says so; CI on
these local commits is stated as pending rather than implied; and the phrasing is
"0 unwaived regressions with 4 provisional accepted shortfalls", never
"0 regressions" -- now moot, since it is 2 unwaived regressions.
Nothing is ratified, relicensed, tagged or published here.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…r profile
The previous commit made the parity comparison judge every dimension
independently and added vertical drift to it, which turned "0 regressions" into
two. Neither was waived, on the grounds that the cause was only *suspected*. It
is now measured, and the two turn out not to be the same failure.
Re-running with refinement off separates them:
f1_fpdf_brief refine 3: dy50 0 -> 1.20 REGRESSION
refine 0: same
-> caused entirely by the closed loop. D2's box-top bias inside
refine(), the mechanism already described in STATUS §7.
05_memo refine 3: dy50 0.59 -> 1.89 REGRESSION
refine 0: still worse
-> upstream of the loop, at parse.
Probing 05_memo directly gives the cleanest statement of D2 in the repository,
because it is the minimal document and nothing else is happening on the page:
PyMuPDF PDFium delta
first 7 baselines identical identical 0.00pt
first line-box top 77.02 78.53 +1.51pt
derived margin_t 77.0 78.5 +1.50pt
Baselines agree exactly, box tops do not, and `infer()` derives the page origin
from box tops -- so the page shifts 1.5pt. That is the same constant already
recorded for 02_research_paper (margin_t 63.30 against 64.90), reached
independently on a different document. 05_memo also groups into 12 blocks against
PyMuPDF's 10, the known grouping divergence, but the drift is the origin.
So both are D2, in its two known locations. All six affected documents are
core-14; every document that embeds its fonts is untouched. They remain UNWAIVED:
six of sixteen excused under one defect is a different product claim from two, and
attribution is not authorisation.
Also fixed, and found by that same refine-0 run: parity floors are measured at one
refine profile and were being applied to any other. 01_whitepaper_market reports
dy_p50 1.39 at refine 3 and 7.89 at refine 0, so the refine-0 run produced four
"below-floor" failures that meant only "these runs are not comparable". The policy
now records the profile its floors were measured at and declines to apply them
across a different one. This landed as a false red; the same mechanism inverted is
a false green.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…chain
The first CI run on this stack went red, and not for anything in the converter.
All three pull requests failed on `c4_i18n`:
regression c4_i18n.pdf dy_p50 0.15 -> 0.7 (recorded 0.15, tolerance 0.5)
The gate REGENERATED its 16 input documents before measuring them. The baseline
was recorded against a corpus built with Chromium 149; GitHub's ubuntu-24.04
runner ships Chromium 150 (`producer=Skia/PDF m150` in the log); `c4_i18n` -- the
CJK/RTL page, exactly where a browser's font fallback moves -- came out a
different document, and a gated metric moved 5x with no change in this
repository. The other fifteen were identical, so this was not noise: it was one
input file being a different file.
The corpus manifest added last commit caught it precisely and said so
("same page count, different document"), which is the check working. But catching
it every time a browser updates is not determinism.
**A generated corpus cannot be a measurement baseline.** The 16 inputs are now
frozen in testkit/fixtures/, pinned by SHA-256, committed -- 563 KB, checkable by
anyone, and the bytes the numbers describe are the bytes in the repository. The
metric gate no longer needs Chromium at all.
The generators are not deleted: tests/test_corpus_generation.py runs them and
checks they still work. It reports drift and gates only on breakage, because
"the toolchain moved" and "the converter regressed" are different questions that
were being asked as one.
That test also demonstrates why freezing was the only option. Regenerating on the
*exact machine* that produced the fixtures changes all 16 SHA-256 hashes:
ReportLab and Chromium stamp a creation time and a document ID into every file.
The corpus was never byte-reproducible, on any machine, ever. So the drift report
uses a content digest over page geometry and normalised text, which carries no
timestamp and moves only when a document really changed -- as c4_i18n did.
Three identity mechanisms, one job each, no overlap:
sha256 of input bytes corpus_manifest are these the exact bytes measured?
content fingerprint generation test did the toolchain change a document?
golden IR digests golden_ir did the parser's output change?
Second half of the same defect: `evidence.environment()` set
`canonical = (os == "linux")`. That was true of the machine the baseline was
recorded on AND of the runner that disagreed with it -- same flag, different
Chromium, different Python patch, different answer. An environment check that
cannot distinguish the environments you actually have is decoration. `canonical`
is now an exact match against a declared toolchain (OS, Python minor, LibreOffice,
the metric fonts, and every measurement dependency), it names what differs rather
than just saying no, and the artifact carries a short fingerprint of it.
Chromium is deliberately NOT in that fingerprint: with the corpus frozen it no
longer touches a measured number. It is recorded for provenance and does not gate.
Verified on the canonical environment with the frozen fixtures: both lanes PASS
reproducing the recorded baseline exactly (product within-2pt 0.5118, dy50 0.62pt,
15/16 page match; raw 0.3486, 2.20pt, 13/16), and parity FAILS on exactly the two
known unwaived regressions and nothing else.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Owner
Author
Update: the first CI run was red for an unrelated reason, now fixedActions failed all three PRs on Fixed by freezing the corpus: the 16 inputs are now committed in Regenerating on the machine that produced the fixtures changes all 16 hashes Verified on the canonical environment: both lanes PASS reproducing the recorded |
Freezing the corpus was necessary and not sufficient. With all 16 inputs
byte-identical by SHA-256, CI still disagreed with the recorded baseline on
c4_i18n, and by more than before:
c4_i18n dy_p50 0.15pt -> 2.1pt
within2pt 0.416 -> 0.038
Same input bytes, same LibreOffice build, same Python minor. The remaining
variable was the font set. c4_i18n is the CJK + Arabic + Hebrew document, and
Liberation covers none of those scripts -- a GitHub runner image ships a large
font collection, so LibreOffice resolved those runs to faces the 59-font
measurement environment does not have.
Installing the right fonts is half the job. Seeing no others is the other half.
scripts/fonts.conf therefore REPLACES fontconfig's search path rather than adding
to it, and every step sets FONTCONFIG_FILE, so the renderer sees exactly:
Liberation metric-compatible with Arial / Times New Roman / Courier New
DejaVu Latin fallback and Hebrew
FreeFont Arabic
WenQuanYi Simplified Chinese
IPA Gothic Japanese
bootstrap.sh installs that set on apt, dnf and apk rather than only the Latin two,
and writes FONTCONFIG_FILE into scripts/env.sh alongside the oracle paths.
**The baseline is re-recorded, and this is not re-recording to clear a failure.**
The pinned set is not the set that happened to be installed: it drops Bitstream
Charter, Courier 10 Pitch, Loma, Unifont and OpenSymbol, which the measurement
container had only because LibreOffice's dependencies pulled them in. So the
canonical environment is now *defined* rather than incidental, the numbers
describe that definition, and they moved:
was now
product <2pt 0.5118 0.4981
raw <2pt 0.3486 0.3349
page match 15/16, 13/16 unchanged
live text 0.9652 unchanged
Nothing about the converter changed. This is the cost of stating an environment
instead of inheriting one, paid once.
Also: `oracle_versions()` recorded only families matching `Liberation \w+`, which
is why two environments with completely different CJK and RTL coverage both
looked identical in the evidence. It now records every visible family and the
count, the canonical check requires the scripts the corpus actually contains, and
an unset FONTCONFIG_FILE is itself a canonical-environment mismatch -- because a
renderer that can see whatever the machine carries is not a pinned renderer.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…-identical Actions now reproduces the recorded baseline exactly on both lanes, which is what the corpus freeze and the font pin were for. The only red step is parity, on the two unwaived dy_p50 regressions and nothing else -- the state this stack is supposed to be in until they are fixed or ratified. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ebt55
changed the base branch from
claude/exactdoc-pr2-backend-seam
to
claude/exactdoc-execution-plan-42d13a
July 30, 2026 10:26
ebt55
changed the title
This was referenced Jul 30, 2026
ebt55
changed the base branch from
claude/exactdoc-execution-plan-42d13a
to
main
July 30, 2026 10:29
INT-00 from the Google-Docs-first build plan. The pre-transition graph, every merge-base and commit count, and the live CI state for PRs #1-#4, captured before anything was mutated so the whole transition is reversible by inspection. Three things the verification found, none of which was in the plan or in the repository: - the plan's own graph draws PR #2 as based on the execution-plan branch. It is based on `main`. Branch ancestry is as drawn; the pull request's declared base is not. The retarget still yields the intended seven commits, verified by merge-base before it was requested and by GitHub after. - the parity gate fails in FIVE places, not two: the two unwaived dy_p50 regressions plus three `c4_i18n` below-floor dimensions. PR #4's body said two, and ROADMAP.md still says the gate fails on exactly the two unwaived regressions and nothing else. The verdict tally and the failure count are different things, and conflating them is how the claim survived. - those three `c4_i18n` floors are labelled `ratified` and are stale. They were recorded before the font set was pinned, and `c4_i18n` is the CJK + Arabic + Hebrew document -- precisely the one a font environment moves. They are to be remeasured, not waived. No force-push, no branch deleted, no public SHA rewritten: all six branches still resolve to their pre-transition commits, and PR #1 was merged with a real merge commit (two parents) so 0cd7d11 remains an ancestor of main. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…and ratified DET-02 from the Google-Docs-first build plan. Two contracts the gate could not actually enforce, and 38 new mutation assertions covering every way each one used to pass. 153 -> 191, still no corpus and no oracle. ## The environment was called canonical on four things that are not tests `environment_identity()` compared the Python *minor*, a LibreOffice version *prefix*, a *subset* of the required fonts, and whether FONTCONFIG_FILE was merely non-empty. Then it computed a fingerprint and never compared it against anything -- it could not have, because `environment()` assigned the fingerprint *after* calling the identity check. Four ways to be a different environment and still report `canonical: true`, and this repository has already been burnt by two of them: Chromium 149 vs 150 moved a gated metric 5x, and an unpinned font set moved c4_i18n's within2pt from 0.416 to 0.038 with the corpus already frozen byte-for-byte. Now every comparison is an equality against a *recorded* canonical run, kept in testkit/canonical_env.json and written by `evidence.py --record-canonical`. Recorded data rather than constants in the source, because the exact LibreOffice build and font set are measurements, and a hand-maintained constant is one edit away from describing an environment nobody ran. Added to the fingerprint: the exact Python version, the exact LibreOffice build string, the SHA-256 of scripts/fonts.conf, a digest over every visible font FILE, and the OCI image digest when a run declares one. Two checks are new in kind rather than just stricter: - the font set must match in BOTH directions. Schema 1 only asked whether the required families were present, so a runner shipping extra faces on top of them read as canonical -- which is exactly what moved c4_i18n's dy_p50 from 0.15pt to 2.1pt. Installing the right fonts is half the job; seeing no others is the other half. - FONTCONFIG_FILE must resolve to this repository's fonts.conf by *content*, not merely be set. A config that is set is not a config that is applied, and pointing it at /etc/fonts/fonts.conf satisfied the old check while adding the system's fonts back. Recording is refused off Linux, without fonts.conf applied, with any oracle or font digest missing, and -- without --force -- over an existing record, because redefining canonical invalidates every baseline and floor bound to the old fingerprint. No canonical reference is committed yet: recording one requires the canonical environment, so until it exists every run honestly reports NOT canonical with the command to fix it. Fail-closed was the point. ## Accepted meant both provisional and ratified, so it meant neither parity_policy.json had one waiver section, `accepted_shortfalls`, whose own note called its four documents RATIFIED while ROADMAP and STATUS called the same four provisional. The executable rule and the written rule disagreed about whether a release was authorised, and nothing in the gate could tell you which it meant. Schema 2 splits it: - `provisional_shortfalls` -- visible, bounded, attributed, and NOT authorising a swap or a release. Sets release_ready=false and fails the run, because "provisional findings never count as a pass" has to be in the exit code or it is not a rule. - `ratified_shortfalls` -- requires ratified_by, ratified_on, issue and review_condition, all four checked. Without a person, a date and a way to expire, "ratified" is an unbounded waiver wearing the word. The four disputed D2 documents move to provisional. Nothing is ratified yet, and `05_memo` and `f1_fpdf_brief` are deliberately in neither section: attribution is not authorisation, and widening a waiver from two documents to four is a product decision, scheduled for the Google Docs checkpoint because Google Docs is the renderer this project actually targets and LibreOffice is a proxy for it. The retired section name is now a hard failure rather than something to guess at. Every floor also carries the environment_fingerprint, corpus_manifest_sha256, measured_commit and profile_id it was measured under. A floor that does not name its environment cannot be told apart from a regression when one changes -- which is not hypothetical: it is the three stale `c4_i18n` floors currently failing CI as though the backend had moved. Those floors are marked as superseded and are to be remeasured, not waived; `record_policy` writes the binding fields, and remeasuring needs the canonical environment. Also fixed: `test_committed_parity_policy_is_wellformed` read the renamed section, found nothing, looped over nothing and passed -- a zero-iteration false green of exactly the kind this suite exists to catch. It now asserts it has something to check, and that neither unwaived regression has quietly appeared in a waiver section. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
DET-02, the two false-green tests. Both reported success while executing none of
the work they exist to do.
## The PyMuPDF-free proof was proving nothing
`_fixture_dirs()` searched `testkit/adv/` and `corpus/pdfs/`. Both are generated,
and both are gitignored. `testkit/fixtures/` -- the 16 committed, SHA-256-pinned
inputs, the only corpus directory that exists in a clean checkout -- was not in
the list.
So on CI and on any fresh clone the search returned nothing, and the caller then
did this:
if not fixtures:
print("no corpus documents found ...")
return 1 if FAILED else 0 # <- 0. success.
The single claim the Apache alpha rests on -- that every code path runs with
PyMuPDF physically absent -- was being made by a test that had converted zero
documents. The import-boundary assertions above it did run, which is why it
looked plausible.
Now: `testkit/fixtures/` is searched first and wins on a filename collision, so a
stale generated copy cannot shadow the frozen input. Zero fixtures is a failure.
A missing capability category is a failure -- a capability this test cannot
exercise is a capability nobody has shown survives the swap. And the conversion
count is asserted against the expected one at the end, because a loop over an
empty list is a loop that reports nothing wrong.
Added `c4_i18n.pdf` to the capability set: CJK, Arabic and Hebrew went through
the permissive parser untested, and the RTL reordering is a PDFium-specific
behaviour, so it is exactly the wrong thing to leave uncovered.
Measured after the fix: 8 of 8 capability fixtures convert, refinement runs
through the permissive path, and the static gdocs profile writes.
## The generator health test discarded its generators' exit codes
`subprocess.run(...)` was called twice with `capture_output=True`, no `check=`,
and no inspection of `returncode`. A generator that raised on every document
produced nothing, and the absences were then printed as "not generated here
(tool absent on this machine)" and passed. The test whose entire subject is
whether the generators still work could not tell a broken generator from an
uninstalled one.
Compounding it, the floor was three pure-Python documents:
{r1_reportlab_report, f1_fpdf_brief, 05_memo} <= produced
Thirteen of sixteen could vanish and this still passed.
Now both processes' exit codes and stderr tails are checked, `--strict` requires
the exact 16-document set and rejects unexpected ones, every output is validated
as readable and non-trivial with the read failure reported, and there is an
explicit assertion that there was at least one document to validate at all.
Non-strict stays permissive on purpose, so a contributor without the optional
oracles still learns something.
Content drift remains a report rather than a failure outside --strict, which is
the existing and correct design: ReportLab and Chromium stamp a creation time
into every file, the corpus was never byte-reproducible, and conflating "the
tooling moved" with "the converter regressed" is what cost three red pull
requests.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
DET-02. `ubuntu-24.04` on GitHub Actions is not a fixed environment, so it cannot
be the definition of one. Its LibreOffice build, its font collection and its
Python patch level all move without anything here changing, and two CI failures
came from exactly that -- both with the corpus already frozen byte-for-byte.
So the canonical environment is now an image, and testkit/canonical_env.json is
the exact toolchain recorded from inside it.
## What the image resolved
base ubuntu:24.04@sha256:4fbb8e6a8395de5a7550b33509421a2bafbc0aab6c06ba2cef9ebffbc7092d90
python 3.12.3
soffice LibreOffice 24.2.7.2 420(Build:2)
chrome Google Chrome for Testing 149.0.7827.55
fonts 35 files, 14 families visible under scripts/fonts.conf
fp cf51ded391ce5ef7d6cc7711388218e72400679de6ea6234d63cca429c82fdcf
Two of those are worth stating plainly. The runner's Python is **3.12.13**; the
image's is **3.12.3**. A minor-version check called those the same environment,
and this repository has a 5x metric move on its record from a comparable
difference. And Chrome 149 is the browser the frozen fixtures were generated
with, where the runner now ships 150 -- the change that made `c4_i18n` a
different document.
The base is pinned by digest, not by tag. Rebuilding may resolve different apt
versions; that is why the *digest* is the pin. A new digest is a new environment,
and adopting one is a deliberate baseline migration, never a side effect of a
rebuild.
## Verified round-trip, not just written
Recording is refused off Linux, without fonts.conf applied, with any oracle or
font digest missing, and without --force over an existing record. Inside the
image, after recording, a fresh run reports canonical. Both mutations fire and
name the cause:
unset FONTCONFIG_FILE ten unexpected families appear -- Bitstream Charter,
Courier 10 Pitch, Loma, Noto Color Emoji, OpenSymbol,
Unifont and friends, all dragged in as LibreOffice
dependencies and all invisible under the pinned config
a different config named as a fonts.conf digest mismatch, not merely as
"a config is set"
The full mutation suite passes inside the image.
Fixed while verifying: `environment_identity(env, ref=None)` could not tell "load
the recorded reference" from "there is no reference". The no-reference test
therefore passed on a machine with no reference file and failed inside the image
that had one -- an environment-dependent test in the module whose whole subject
is environment dependence. `ref` omitted now loads; `ref=None` states there is
none.
## Publishing is deliberate and has not happened
`.github/workflows/gate-image.yml` is workflow_dispatch only, tagged by commit
SHA with no `latest`, and prints the digest to pin. Nothing is published until
someone triggers it.
Until it is published and gate.yml is pinned to its digest, CI keeps running on
the runner and now reports NOT canonical with the specific differences named
rather than a vague pass. That is the correct state and not a regression: the
runner genuinely is not this environment, baseline recording was already refused
off-canonical, and the gap is now visible instead of assumed away.
The parity floors are still unmeasured in this environment and say so -- every
one reports `unbound`. Remeasuring them is the next step and is a separate
commit, because a floor and the environment that produced it must not move
together silently.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`backend_parity.py --update-policy` raised `NameError: name 'refine' is not
defined` before writing anything. `record_policy` referenced `refine`, which
exists only as a local in `main` -- so the one command `parity_policy.json`'s own
note instructs you to run ("Record the floors with `backend_parity.py
--update-policy` on the canonical environment") crashed on every invocation.
This is the mechanical explanation for something that had been read as neglect.
The three stale `c4_i18n` floors were never remeasured after `scripts/fonts.conf`
pinned the visible font set, and the reason is not that nobody tried -- it is that
trying crashed before touching the file. A gate whose recording path is unrunnable
can only ever compare against floors from whenever the path last worked.
Found by running it, in the canonical image, which is the first time it has been
run since the line was introduced. The fix is to pass the measured profile in
rather than reach for it: `record_policy(..., refine=refine)`.
Now tested. `test_parity_policy_can_actually_be_recorded` calls the real recording
path against a temp file and asserts it (a) runs at all, and (b) writes floors
that name their environment, corpus, profile and refine rounds. A recording path
with no test is a recording path nobody has executed, and this suite exists
precisely to make that impossible to sustain. 191 -> 196 assertions.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
DET-02's last measurement. Floors recorded in the canonical image (fingerprint
cf51ded391ce5ef7...), full 16-document corpus verified byte-identical to the
manifest first. Before/after diff, as plan §17 rule 3 requires of any re-record:
c4_i18n.pdf doc_recall 0.9874 -> 0.9748
dy_p50 0.15 -> 0.8
within2pt 0.5745 -> 0.3017
word_recall 0.9874 -> 0.9748
01_whitepaper_market, 02_research_paper, 03_tech_report_code,
r1_reportlab_report, c5_graphics floors unchanged
**`c4_i18n` was the only document whose floors moved.** The other five came out
bit-identical on every dimension. That is what turns the font environment from a
plausible explanation into an attributed one: `c4_i18n` is the CJK + Arabic +
Hebrew page, Liberation covers none of those scripts, and a change confined to
exactly the document that depends on the variable is not a coincidence. Had the
cause been the backend, the core-14 documents would have moved too.
Every floor now names its environment, corpus and profile, so the next
toolchain change is detectable instead of arriving disguised as a regression.
`measured_commit` is stamped from the checkout and says so: the image held a copy
of the tree with no git repository, which `evidence.git_state()` correctly
reported rather than guessed at. The authoritative re-record comes from CI once
gate.yml runs inside the published image.
## Where the gate now stands
2 regression(s), 5 same, 3 better, 2 expected-divergence, 4 accepted
regression 05_memo.pdf worse on dy_p50: 0.59 -> 1.89
regression f1_fpdf_brief.pdf worse on dy_p50: 0 -> 1.2
provisional 01_whitepaper_market.pdf
provisional 02_research_paper.pdf
provisional 03_tech_report_code.pdf
provisional r1_reportlab_report.pdf
FAIL
Six failures. Zero `below-floor`, zero `unbound`. Across this work the count went
5 -> 15 -> 6 and the converter did not move once: 5 understated the truth, 15 was
the honest count once floors had to name their environment, and 6 is what is left
after the environment debt was paid. Both remaining classes are product decisions
rather than measurement debt, and the four provisional ones are deliberately
scheduled for the Google Docs checkpoint.
ROADMAP.md is corrected in the same commit, per plan §17 rule 12. It had claimed
the gate "fails on exactly the two unwaived regressions and nothing else", which
its own CI run disproved.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The canonical reference recorded in the previous commit was invalid on Linux,
which is the only place it has to work.
`scripts/fonts.conf` is hashed into the environment fingerprint, so its bytes are
a gated value rather than configuration. `.gitattributes` pinned `*.sh`, `*.yml`
and `*.yaml` to LF but not `*.conf`, so a Windows checkout produced CRLF. The
reference was recorded from a copy of that checkout and hashed
924510e8... (CRLF, this machine's working tree)
84d4357a... (LF, every Linux checkout -- CI included)
CI reported it accurately and the message was correct in every particular; it was
the record that was wrong. The committed file was never CRLF -- git stores it LF
-- so the defect existed only in the path from a Windows working tree into the
measurement container, which is exactly the path a digest cannot check about
itself.
## Re-recorded, and the floors did not move
New canonical fingerprint `3ca438f17d905cef...`, recorded against the LF bytes.
All six waived documents rebound to it, and the check that matters:
floor values moved: 0
Every floor on all six documents came out identical. That is the evidence that
this was an identity defect and not a measurement one: CRLF does not change how
an XML parser reads a fontconfig file, so the renderer saw the same font set and
produced the same numbers under both spellings. Asserted rather than assumed --
the re-record compared before and after rather than declaring the difference
cosmetic.
Parity is unchanged at six failures: the two unwaived `dy_p50` regressions and
the four provisional D2 shortfalls. Zero below-floor, zero unbound, zero
environment-mismatch inside the image.
## Guarded
`test_hashed_files_have_one_spelling` asserts that every file whose bytes reach
the fingerprint is checked out LF-only, AND that `.gitattributes` carries the rule
that keeps it so. The second half matters: a Linux-only CI would pass the byte
check while a Windows contributor kept recording unreproducible digests, so
checking the bytes alone would have been a test that only fails where the bug
cannot happen. 196 -> 198 assertions.
Note this does not by itself make CI canonical. The runner still differs on
Python (3.12.13 against the image's 3.12.3) and ships four extra DejaVu variants,
and it reports both. Those close when gate.yml runs inside the published image,
not before.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
REL-01A, first slice. Three new modules and their tests, wired into nothing yet: pure addition, so no recorded fidelity value can move. Wiring `convert()` onto them is the next commit, where the invariant has to be proved rather than assumed. ## exactdoc/io.py -- publication is transactional `doc.save(out_path)` writes straight to the destination, and python-docx serialises a ZIP incrementally. Any exception partway through -- a figure that fails to rasterise, a full disk, a cancelled run -- left a truncated file where a good document used to be. The caller asked for a new version and lost the old one. Publication now serialises into a unique temp file **in the destination directory**, validates it, fsyncs, and `os.replace`s. Same directory because `os.replace` is only atomic within a filesystem: a temp file in /tmp crossing a mount degrades to a copy, which is the non-atomic write this exists to prevent. Validation runs *before* replacement, and it is structural rather than semantic: readable ZIP, `[Content_Types].xml`, `word/document.xml` present and non-empty, `_rels/.rels`. An unreadable DOCX that replaced a readable one is not better than a failed conversion, and the writer cannot always tell it produced one. `Workspace` replaces the refinement loop's predictable adjacent `<dest>.best` path. That name meant two conversions of the same input in one directory overwrote each other's candidates, and a crash left the intermediate lying around looking like a deliverable. `tests/test_atomic_output.py` injects a failure at each stage -- writer raises mid-write, output is not a ZIP, ZIP is missing `word/document.xml`, document body is empty, output is zero bytes -- and asserts the destination is **byte-identical** every time. Injection rather than mocking the path, because the property is what happens when real code raises and a mock that never raises proves nothing. Also asserted: no temp file survives success or failure, twelve conversions to one destination take twelve distinct temp names, and the workspace is removed even when the conversion throws. ## exactdoc/errors.py -- what went wrong, as a type Failures surfaced as whatever the layer beneath happened to raise: `ValueError: document closed or encrypted` for a password-protected PDF, a `KeyError` from inside the writer for a malformed one, and -- worst -- nothing at all when a requested renderer was missing, because the conversion silently ran open-loop. That last one shapes the hierarchy. **A missing oracle is not a degraded success.** `OracleUnavailableError` is an error, not a warning; producing an unrefined DOCX when refinement was requested is a different product reported under the same exit code, and it is how a published fidelity number came to describe a profile no surface ran. `CloudConsentRequiredError` is deliberately a *configuration* error: nothing has been attempted and nothing uploaded. "The target is gdocs" must never on its own mean "you may send this document to a third party" -- that conflation is the privacy-dangerous ambiguity the profile/oracle split exists to remove. `OracleCleanupError` documents that a failed remote delete is a privacy failure, not a tidiness one: the caller's content is still in someone else's storage. ## exactdoc/result.py -- requested vs resolved, always `convert()` returned a path, which is enough to find the file and not enough to trust it. The result now carries requested and resolved options side by side, and `degraded` answers the question the old contract could not: did I get the conversion I asked for? Everything in it is content-safe -- hashes, counts, durations, option names; no source text, credentials, or remote identifiers. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
REL-01A. `target` was answering two independent questions, and the consequence
was not cosmetic: there was no way to ask for **Google-Docs-safe OOXML produced
offline**, which is the configuration this project intends to ship. Wanting
Docs-shaped output implied wanting to upload the document to Google.
output_profile how the OOXML is written. Pure serialisation: offline,
deterministic, no network, no credentials.
oracle what renders the result during refinement. Only consulted
when refine_rounds > 0; `none` is a real answer.
## Behaviour-preserving where it must be
`PRODUCT` is `standard` + `libreoffice`, which is exactly what
`target="libreoffice"` meant, and `line_mode_for` keys on the profile with the
same `== "gdocs"` test as before. The same documents get the same line-height
encoding, so no recorded fidelity value can move. Asserted rather than assumed:
22 checks on the split, and both lanes re-measured in the canonical image.
`RAW` now says `oracle="none"` as well as `refine_rounds=0`. Naming a renderer it
would never call made the control lane look like it had one.
## Two silent degradations, now errors
`convert()` asked for a renderer, got `None`, and fell through to an open-loop
write -- printing a line only under `--verbose`. A caller who asked for
refinement received an unrefined document and a success exit code. That is the
mechanism by which a published fidelity number came to describe a profile no
surface ran, and `get_renderer` returning `(None, "none")` for an absent
LibreOffice is the same failure one layer down. Both now raise
`OracleUnavailableError`. `refine_rounds > 0` with `oracle="none"` is refused at
construction, before any output exists.
## Formatting is not consent
`target="gdocs"` used to mean both "write Docs-safe OOXML" and "upload this
document to Google", inseparably. It now selects the **profile** only. The cloud
oracle requires `allow_cloud_upload=True` per call, or `--allow-cloud-upload`,
and it is deliberately never read from the environment: an exported variable must
not be able to authorise sending somebody's document to a third party. Choosing
`output_profile="gdocs"` requires no consent because it contacts nothing.
`target=` still works for one alpha cycle with a `DeprecationWarning` that spells
out the mapping. Conflicting legacy and new arguments are refused rather than
silently merged, and `--target` defaults to unset so it cannot outrank the new
pair on every invocation.
## A third zero-input false green
`tests/test_purity.py` searched only `corpus/pdfs/` and `testkit/adv/`, both
generated and both gitignored. Its pytest entry point asserted the corpus was
non-empty; its `__main__` did not -- and `gate.yml` runs `__main__`. On a clean
checkout it printed "0/0 documents reproducible" and exited 0: a write-purity
proof that had written nothing twice. Now searches the frozen fixtures first and
fails on zero. Measured after the fix: 16/16.
That is the third instance of this shape in three files. The pattern is always
the same -- a test whose inputs are generated, in a repository whose inputs are
frozen.
202 assertions in the mutation suite, 24 in the atomic-output suite, all green.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…a diff that compares nothing
REL-01A's invariant needed a way to be checked, and the gate cannot do it. The
gate passes within a tolerance, which is right for "has the converter
regressed?" and wrong for "did this refactor change layout?" -- a value that
moves by less than the tolerance has still moved, and a change whose whole claim
is neutrality should be held to bit-equality.
`testkit/exact_baseline_diff.py` compares every recorded value with `==`. Run
after the lanes; exit 0 means nothing moved, exit 1 names what did.
Exit **2** means it compared nothing, and that is deliberately not a pass. The
first version of this comparison reported "32 MOVED" against a run that had in
fact reproduced the baseline perfectly -- it had failed to key into
results.json, compared zero values, and every recorded metric came back MISSING.
A diff that inspects nothing reports the same shape of result as a catastrophic
regression, and it took reading the output twice to tell them apart. That is the
same false green found in three test files this week, in a fourth costume.
`page_err` is derived rather than stored: results.json carries `out_pages` and
`src_pages`, the baseline records the difference. Skipping absent keys quietly
would have dropped precisely the metric that catches a pagination change, which
is the largest open defect class in this project. So the derivation is explicit
and named.
## The result it was built to produce
compared 224 recorded values across 2 lanes
ZERO MOVED -- every recorded value reproduced exactly
2 lanes x 16 documents x 7 metrics, in the canonical image, against the recorded
baseline. The profile/oracle split changed no measured value, which is what G1
promised and had until now only asserted.
Both lanes also pass their own gate with the recorded aggregates unchanged --
product 15/16, 0.4981, 0.9652, 0.675; raw 13/16, 0.3349, 0.9652, 2.2 -- and the
parity verdict is the same six failures as before the refactor.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… it will never do The README described a general PDF->DOCX converter with a `target` option, one value of which happened to be Google Docs. That is not the product. Docs is the target, the licence is the adoption blocker, and everything else is detail. Rewritten around those two, with the numbers as they actually stand. ## Removed: two documents that describe a tool that no longer exists - `SESSIONS.md` (85 KB) -- a per-session working log written to an internal protocol, last touched seventeen commits ago. Its value was process discipline during a phase that has ended; as the largest file in a repository someone is meant to evaluate, it is noise. - `FINDINGS.md` (18 KB) -- opened by declaring itself "a frozen audit of v1.1 as it was found, not a description of the tool today", against an 18-document corpus that no longer exists, with two claims already marked as falsified. A reader landing there learns about a version that never shipped. Their surviving content -- root causes, the defect register, the design reasoning -- is in STATUS.md and THEORY.md, which are maintained. One code comment in `parse_pdfium.py` cited SESSIONS.md for why the sliding-window pitch estimate was reverted; it now states the reason itself rather than pointing at a file, which is what it should have done in the first place. ## Three kinds of limit, told apart The old README mixed "we will fix this", "this is impossible" and "this is possible but not worth it" under two headings. They deserve different answers, and the third one was missing entirely: **Dialects that will stay hard -- where a fallback beats a fix.** Heavy LaTeX/pdfTeX, whose vertical model is built on glue and penalties that OOXML cannot express, and highly designed pages that were never flow documents. These are not "not yet supported"; chasing them to pixel fidelity means reimplementing a typesetting engine, and the return curve is bad -- three attribution attempts on the LaTeX pagination defect each produced a partly-wrong answer. The pragmatic treatment is to **rasterise the problematic region and keep the surrounding text live**. A page that is 90% editable with one faithful image of an un-modellable figure beats a page that is 100% "editable" and visibly wrong, and beats the whole-page rasterisation most converters fall back to. The converter already does this for gradients and vector artwork; doing it deliberately for these dialects, with a reported budget, is the intended direction. It also says plainly that if your documents are mostly LaTeX papers or design-led pages, this is the wrong tool today. That is cheaper for both sides than letting someone find out. ## Also corrected The Google Docs column is now labelled exploratory rather than presented beside the LibreOffice numbers as though comparable: the two come from different corpora and no manifest-bound same-run gate produced them. The 0/4 holdout leads rather than hides. The licence section explains that the AGPL is inherited from PyMuPDF rather than chosen, that every path already runs with PyMuPDF absent, and that what remains is a Google Docs measurement and a legal review rather than code. ## And a gap the rewrite exposed Writing the usage example revealed that `--oracle none` with the default `--refine 3` raises -- correctly, since refinement without a renderer is not refinement -- but reached the user as a **traceback**. That is the right output for a bug and the wrong one for an ordinary, user-fixable mistake, and it buries the actionable sentence under a stack. `cli.main` now catches `ExactdocError`, prints the message and its detail to stderr, and returns a documented exit code: 3 config, 4 cloud-consent-required, 5 unsupported-input, 8 output-write, 11 oracle-unavailable, and so on. A script branching on exit status is an API whether or not anyone called it one. 130 on interrupt. Verified: the corrected README command exits 0, `--oracle none` exits 3 with one readable line, `--oracle gdocs` without consent exits 4. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…nd why
A visitor arriving at a red badge cannot tell "this build is broken" from "this
gate is refusing to lie", and the difference is the entire premise of the
project. So the README states it up front, before Install.
Twelve failures, one step, two kinds:
6 decisions nobody has made yet -- 2 unwaived regressions and 4 provisional
shortfalls, all attributed to one cause (PDFium substitutes a generic font
ascent where PyMuPDF reads the real one from an AGPL table). Widening a
waiver from two documents to six is a product decision, and the policy is
built so an executor cannot make it by editing a file.
6 the CI runner not being the reference environment. Every floor records the
environment that produced it; the runner is not the digest-pinned
container, so the gate refuses to compare against numbers describing
somewhere else. Nothing here reflects converter quality, and it clears when
CI runs inside the published image.
Also stated: both fidelity lanes pass with all 224 recorded values reproduced
exactly, so the converter is not regressing -- the gate is waiting on a decision
and an environment, and is being explicit about which.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ebt55
force-pushed
the
claude/exactdoc-pr3-gate-hardening
branch
from
August 1, 2026 21:45
4629d58 to
92a6f5d
Compare
ebt55
changed the title
ebt55
marked this pull request as ready for review
August 1, 2026 21:46
ebt55
added a commit
that referenced
this pull request
Aug 1, 2026
…gate on purpose PR #4 is merged (ef653c9, 19 commits) and the execution log said it was still a draft. Rule 12 applies to this file as much as to the README: a record that describes the state before the change is worse than no record, because it reads as current. Also records what the merge cost and did not cost: every feature branch was verified reachable from main before deletion, the remote is now main alone, and there are no open PRs or issues. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
This PR strengthens the project’s “production foundations” by hardening the fidelity gate (fail-closed behavior, mutation tests, two-lane gating), formalizing a frozen/verified corpus + pinned font environment, and refactoring the conversion API around explicit, validated options (output profile vs render oracle) with typed errors and a backend seam that supports running without PyMuPDF.
Changes:
- Introduces
ConversionOptions/profiles (PRODUCT,RAW) and splits legacytargetintooutput_profile(serialization) vsoracle(refinement renderer), with typed error hierarchy + CLI exit-code mapping. - Adds corpus determinism infrastructure: SHA-256–pinned fixtures + manifest tooling, generator-health test, and pinned fontconfig environment (plus canonical environment records / image).
- Reworks gate runner and CI workflow to run both lanes, verify corpus identity, and fail closed; adds additional tests and diagnostics scripts.
Reviewed changes
Copilot reviewed 52 out of 69 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| THEORY.md | Updates theory narrative around lane semantics and default profiles. |
| tests/test_purity.py | Fixes corpus discovery to include committed fixtures; fails if zero inputs. |
| tests/test_no_pymupdf.py | Adds hostile “fitz unimportable” test validating permissive runtime boundary. |
| tests/test_corpus_generation.py | Adds generator health test with exit-code checks and drift reporting. |
| tests/test_atomic_output.py | Adds failure-injection tests for atomic publishing behavior. |
| testkit/runall.py | Refactors batch runner to use manifest corpus, lane profiles, and gate.py decision. |
| testkit/README.md | Documents hardened gate model, determinism approach, and tooling roles. |
| testkit/gen_corpus.py | Adds --strict mode to make generator skips fatal in CI contexts. |
| testkit/exp_sweep.py | Updates experiment to use explicit zero-refine profile via options. |
| testkit/exp_regroup.py | Updates experiments to register parsers via backend seam (no global patching). |
| testkit/exp_chromefix.py | Same: registers experimental backend instead of patching exactdoc.convert. |
| testkit/exact_baseline_diff.py | Adds exact-value baseline diff tool (fail on “compared zero values”). |
| testkit/edge_cases.py | Makes edge-case conversions explicitly use RAW profile (no refine dependency). |
| testkit/corpus_manifest.py | Adds manifest verification/update logic for frozen fixtures + content fingerprinting. |
| testkit/corpus_manifest.json | Adds the pinned 16-document corpus manifest (hashes, metadata, fingerprints). |
| testkit/canonical_env.json | Records canonical environment identity (deps, fonts, versions, fingerprint). |
| testkit/backend_superscript.py | Adds measurement script for superscript flag relevance across backends. |
| testkit/_paths.py | Makes oracle discovery read scripts/env.sh directly (no “source it” assumption). |
| scripts/fonts.conf | Pins renderer-visible font search path and substitutions via fontconfig. |
| scripts/bootstrap.sh | Installs required fonts/tooling; uses uv sync --frozen; records FONTCONFIG_FILE. |
| exactdoc/verify.py | Removes module-scope fitz; routes rasterization + parsing through backend seam. |
| exactdoc/targets.py | Reworks renderer selection around oracles; errors on unavailable requested oracle. |
| exactdoc/result.py | Introduces structured conversion result model (requested vs resolved, warnings, oracle runs). |
| exactdoc/refine.py | Routes rendered/source line extraction through backend seam; errors when refinement requested without oracle. |
| exactdoc/parse_pdfium.py | Closes native PDFium handles properly; fixes web-links handle lifecycle. |
| exactdoc/options.py | Adds validated immutable options, profile defaults, legacy target migration mapping. |
| exactdoc/metrics.py | Adds permissive text-metrics boundary (Null vs MuPDF metrics), used by ladder. |
| exactdoc/ladder.py | Refactors ladder shaping to depend on injected metrics; makes “unmeasurable” explicit. |
| exactdoc/io.py | Adds transactional DOCX publish + structural validation + scratch workspace helper. |
| exactdoc/errors.py | Adds typed error hierarchy + stable error codes. |
| exactdoc/docxout.py | Removes module-global writer state; threads write context + figure rendering via backend seam. |
| exactdoc/convert.py | Refactors convert() to resolve full options profile + explicit backend/oracle/profile behavior. |
| exactdoc/cli.py | Builds new CLI with explicit oracle/profile/backend flags + stable exit codes. |
| exactdoc/backend.py | Expands backend protocol (rendering + page line extraction) and adds backend registration. |
| exactdoc/init.py | Adds lazy exports and keeps import lightweight for permissive-runtime goals. |
| docs/evidence/pr-transition.md | Adds PR transition record and CI-state evidence. |
| docs/evidence/execution-log.md | Adds execution log for plan progress and standing constraints. |
| docker/gate.Dockerfile | Adds canonical measurement image definition pinned by digest. |
| .gitignore | Extends ignore patterns for plan docs. |
| .github/workflows/gate.yml | Hardens CI gate: pinned fonts, fixture verification, new tests, fail-closed lane gating, parity no longer continue-on-error. |
| .github/workflows/gate-image.yml | Adds manual workflow to publish canonical gate image by digest. |
| .gitattributes | Pins LF for conf files and treats frozen fixtures as binary to preserve hashes. |
Suppressed comments (1)
exactdoc/convert.py:123
- The
__main__path callsmain()but does not propagate its return code, sopython -m exactdoc.convertwill exit 0 even when the CLI fails. Sincemain()is now a thin wrapper returning the real CLI exit status, it should be returned as the process exit code.
if __name__ == "__main__":
main()
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+111
to
+113
| from .docxout import write_docx | ||
| return write_docx(lay, out_path, dpi=opts.dpi, | ||
| output_profile=opts.output_profile, backend=bk) |
Comment on lines
+815
to
+819
| def render_clip(page_no, clip, at_dpi, _bk=backend, _p=lay.src_path): | ||
| try: | ||
| return _bk.render_clip(_p, page_no, clip, dpi=at_dpi) | ||
| except Exception: | ||
| return None |
Comment on lines
+59
to
+69
| import corpus_manifest | ||
| problems, paths = [], [] | ||
| for kind, doc, why in corpus_manifest.verify(manifest): | ||
| problems.append((kind, doc, why)) | ||
| bad = set(d for k, d, _ in problems if k in ("missing", "identity", | ||
| "unmeasured", "duplicate")) | ||
| for doc_id in sorted(manifest.get("documents", {})): | ||
| if doc_id in bad: | ||
| continue | ||
| paths.append(corpus_manifest.fixture_path(doc_id, manifest)) | ||
| return paths, problems |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Production foundations: a gate that can fail, a permissive backend seam, an
exact environment, and a conversion API that cannot quietly do the wrong thing.
19 commits. Merging red, deliberately — see Why this merges red below.
What this contains
fitzunimportable and converting real documentsscripts/fonts.confreplaces fontconfig's search path; installing the right fonts is half the job, seeing no others is the other halfoutput_profilesplit fromoracleThe five false greens
Each was a way for a check to pass while proving nothing. Four were named in the
audit; the fifth was found by running the thing.
environment_identitycompared a Python minor, a LibreOffice versionprefix, a subset of fonts and a merely non-empty
FONTCONFIG_FILE—then computed a fingerprint it never compared against anything. It could not
have:
environment()assigned the fingerprint after calling the check.accepted_shortfallsmeant both "provisional" and "ratified" — its ownnote said RATIFIED while every prose document said provisional, so the
executable rule and the written rule disagreed about whether a release was
authorised.
test_no_pymupdf.pysearched only gitignored directories, found zerodocuments on any clean checkout, and returned 0. The single claim the Apache
relicence rests on was being made by a test that had converted nothing.
test_corpus_generation.pydiscarded both generators' exit codes andrequired 3 of 16 documents, so a totally broken generator was
indistinguishable from a healthy run.
backend_parity.py --update-policyhad never worked. It raisedNameErrorbefore writing anything — which is the mechanical reason threestale
c4_i18nfloors were never remeasured after the font set was pinned.Not neglect: the documented command crashed.
Plus a sixth of the same shape in
test_purity.py, and a seventh in thebaseline-diff tooling itself, which reported "32 MOVED" against a run that had
reproduced the baseline perfectly — it had compared zero values.
REL-01A: two axes, and two silent degradations
targetanswered two independent questions, and the consequence was notcosmetic: there was no way to ask for Google-Docs-safe OOXML produced
offline, which is the configuration this project intends to ship. Wanting
Docs-shaped output implied uploading the document to Google. Now
output_profiledecides serialisation andoracledecides the render loop;--oracle gdocsneeds--allow-cloud-uploadper call, and no environmentvariable can grant it.
Two paths that used to degrade in silence are now errors: a requested renderer
that is missing, and
refine_rounds > 0with no oracle. Both previouslyproduced an unrefined document under a success exit code — the mechanism by
which a published fidelity number came to describe a profile no surface ran.
Fidelity did not move: 224 of 224 recorded values reproduced exactly
(2 lanes × 16 documents × 7 metrics), verified in the canonical image with
testkit/exact_baseline_diff.py.Why this merges red
The parity step fails in 12 places. Every other step passes.
shortfalls, all attributed to one font-metric cause. Widening a waiver from
two documents to six is a product decision, and the policy is deliberately
built so an executor cannot make it by editing a file. They get decided
against Google Docs evidence, because Docs is the target.
the environment that produced them, and the runner is not the digest-pinned
container. Fail-closed working as designed; clears when CI runs inside the
published image.
gate.ymlcarries an explicit instruction not to re-addcontinue-on-error.Going green by ignoring the result is how this gate stopped working once before.
The README's Why CI is red, on purpose section says all of this to anyone who
lands on the badge.
🤖 Generated with Claude Code