Cross-harness context-compaction provenance (kind v1.2.0) - #108
Draft
benbaarber wants to merge 2 commits into
Draft
Cross-harness context-compaction provenance (kind v1.2.0)#108benbaarber wants to merge 2 commits into
benbaarber wants to merge 2 commits into
Conversation
|
🔍 Preview deployed: https://285f33ff.toolpath.pages.dev |
ben-emp
force-pushed
the
ben/compaction
branch
3 times, most recently
from
June 26, 2026 20:48
e0d1bab to
7cb45a7
Compare
ben-emp
force-pushed
the
ben/derive-dedup
branch
2 times, most recently
from
July 1, 2026 17:12
1130be3 to
bc7e3b8
Compare
benbaarber
marked this pull request as draft
July 23, 2026 17:37
ben-emp
pushed a commit
that referenced
this pull request
Jul 23, 2026
Extracted from #108 as its structural prerequisite — the items IR without any compaction content. - ConversationView.turns/.events (two parallel vecs) become one ordered items: Vec<Item> (Turn | Event), preserving the source stream's exact interleaving. turns()/events() stay as iterators. - derive_path emits conversation.event steps for events (previously dropped — a Claude session lost its attachments and system entries on import/export), resolves duplicate step ids by content-aware renaming instead of erroring, and splices events onto the head's ancestry so they don't dangle as false dead ends. extract_conversation restores items, spliced parents, session files_changed, and synthetic-model actors. - Every provider's to_view now builds the ordered item stream. Native compaction markers stay opaque (a generic event, or pi's System turn) — typed boundaries are the follow-up's concern. - The claude projector emits events inline at their item position; real Claude interleaves attachments and system entries with turns, and the trailing-pass writer regrouped them at the end of the file. wire_order_roundtrip pins the projected entry-type sequence to the captured fixture's, and the caveat entry keeps isMeta. - Property tests: unique derived step ids, derive → extract → derive stability, and byte-identical replay dropping, over randomized turn/event interleavings with id collisions, dangling parents, and file mutations.
ben-emp
pushed a commit
that referenced
this pull request
Jul 27, 2026
Extracted from #108 as its structural prerequisite — the items IR without any compaction content. - ConversationView.turns/.events (two parallel vecs) become one ordered items: Vec<Item> (Turn | Event), preserving the source stream's exact interleaving. turns()/events() stay as iterators. - derive_path emits conversation.event steps for events (previously dropped — a Claude session lost its attachments and system entries on import/export), resolves duplicate step ids by content-aware renaming instead of erroring, and splices events onto the head's ancestry so they don't dangle as false dead ends. extract_conversation restores items, spliced parents, session files_changed, and synthetic-model actors. - Every provider's to_view now builds the ordered item stream. Native compaction markers stay opaque (a generic event, or pi's System turn) — typed boundaries are the follow-up's concern. - The claude projector emits events inline at their item position; real Claude interleaves attachments and system entries with turns, and the trailing-pass writer regrouped them at the end of the file. wire_order_roundtrip pins the projected entry-type sequence to the captured fixture's, and the caveat entry keeps isMeta. - Property tests: unique derived step ids, derive → extract → derive stability, and byte-identical replay dropping, over randomized turn/event interleavings with id collisions, dangling parents, and file mutations.
…1.2.0) Restacked onto ben/items-ir (#144) with the standalone fixes split to ben/compaction-pre-split. The Compaction item + the conversation.compact step type (kind v1.2.0): a context-compaction boundary recorded as its own step between the turns it separates, with trigger, summary, pre_tokens, and the anchor-based kept contract (kept_from + expand_kept; the wire carries the expanded contiguous run). Compaction provenance is a closed typed set — no catch-all extra; native detail richer than the typed fields is deliberate loss. Per-provider support, each verified against the real harness TUI by resuming projected sessions (versions in docs/agents/tui-drift-check.md): - claude: compact_boundary + synthetic summary → typed boundary and back, preservedMessages as the kept run, transcript-only + isMeta render flags, empty-turn thinking-block projection. - codex: compacted marker → wholesale boundary (2026-07 payload: encrypted summary + prefix-keep documented as deliberate loss); projector emits the event_msg/context_compacted marker the TUI renders, with native turn_context placement. - opencode: per-boundary summary pairing, compaction-host parent redirects, tailStartID → kept anchor. - pi: compaction entries with parent-chain kept expansion; model_change/thinking_level_change/label as typed events with the chain's model context threaded into projected assistant messages. - copilot: the observed 1.0.68 session.compaction_start/complete pair mapped both directions (wholesale + summary). - gemini/cursor: documented no-boundary. Verification: executable round-trip oracle (testing::{check_view_invariants, assert_fixpoint}), compaction proptests, real captured compacted sessions for claude/codex/pi/ copilot/opencode as fixtures with looped stability tests, the cross-harness compaction matrix (kept-anchor capability flags), the env-gated real-session matrix mode, and import-time event-type surfacing.
derive: byte-identical boundary replays now skip at the source level like turns and events. A replayed compact_boundary (a continuation file repeats its parent's boundary verbatim) was compared on resolved bytes, so any intervening splice or rename made it rename-keep as c#2 - stealing the head, false-dead-ending the real boundary's subtree, and re-pointing later parents at the ghost. The claude provider's dedup exemption explicitly relies on this collapse; it is now real, and a regression test pins the spliced-replay shape. copilot: events and boundaries sharing a turn watermark drained events-first, placing the post-compaction system.message re-prime before the boundary (the real 1.0.68 capture exhibits this). The two position-tagged vecs are now one arrival-ordered Pending stream, the codex shape; ordering pinned on the real fixture and on adjacent boundaries. The cross-harness matrix now sources copilot's compacted fixture. claude: projected boundary/summary entries carry the invariant wire constants (content, level, isMeta, userType, explicit null parentUuid) and cwd/gitBranch parity with turns; the dedup comment cites the observed re-emission block documented in session-chains.md instead of calling it unobserved; the orphan-summary limitation is stated; a tautological wire_order assert now binds. opencode: a kept anchor with no projected message resolves forward (keep-more) instead of writing a dangling raw id into tailStartID; the real capture's summary:true assistant flag survives projection via the typed summary fold; an empty summary.body closes the pending-boundary window instead of extending it; summary messages carrying other parts emit turns; gen-2 derive stability pinned on the real fixture. pi: a kept anchor naming a folded toolResult resolves to the containing assistant turn (keep-more) instead of the next turn. md: conversation.compact steps render at both detail levels, so path show marks where compaction happened; snapshot coverage added. versions: toolpath 0.8.0, toolpath-md 0.8.0, toolpath-git 0.7.0, toolpath-github 0.7.0, toolpath-dot 0.6.0 across all four surfaces - the branch changed toolpath's public constants and md's rendering without bumping, and the changelog already claimed these numbers. docs: changelog entries no longer claim work that lives on the local followups branch (claude stream-parse, codex fresh resume ids); the 06-22 combined entry is marked as superseded narrative and its stale cursor claim fixed; entry date order restored; cursor.md describes the marker round-trip that actually ships; codex.md window_id example matches the wire; the v1.2.0 kind page includes copilot and states the always-stamped event keys.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #144 (
ben/items-ir— the items IR refactor). The standalone fixes were split out to #143. The pre-restack 43-commit history is preserved atben/compaction-pre-split; merging main + #143 + #144 + this PR reproduces that tree byte-for-byte (verified).Compaction in the IR (kind v1.2.0)
The
Compactionitem and theconversation.compactstep type: a context-compaction boundary recorded as its own step between the turns it separates, withtrigger,summary,pre_tokens, and the anchor-based kept contract (kept_from+expand_kept; the wire carries the expanded contiguous run, oldest first). Compaction provenance is a closed typed set — no catch-allextra; native detail richer than the typed fields is deliberate, documented loss.Per-provider support
Each verified by resuming projected sessions in the real harness TUI (versions logged in
docs/agents/tui-drift-check.md):compact_boundary+ synthetic summary ↔ typed boundary,preservedMessagesas the kept run, transcript-only +isMetarender flags, empty-turn thinking-block projection (Claude 2.1.216's renderer aborts on empty text blocks).compactedmarker → wholesale boundary; the 2026-07 payload (encrypted summary inreplacement_history, prefix-keep) documented as deliberate loss; projector emits theevent_msg/context_compactedmarker the TUI renders, with nativeturn_contextplacement.tailStartID→ kept anchor.model_change/thinking_level_change/labelas typed events, with the chain's model context threaded into projected assistant messages (model restore verified in pi 0.72).session.compaction_start/completepair mapped both directions (wholesale + summary).Verification
Executable round-trip oracle (
testing::{check_view_invariants, assert_fixpoint}), compaction proptests, real captured compacted sessions for claude/codex/pi/copilot/opencode checked in as fixtures with looped stability tests (kept-run expansion was hash-order-flaky at ~1 run in 4 before the determinism fix), the cross-harness compaction matrix with kept-anchor capability flags, the env-gated real-session matrix mode, and import-time event-type surfacing.