Unify turns and events into ConversationView.items - #144
Draft
benbaarber wants to merge 10 commits into
Draft
Conversation
ben-emp
pushed a commit
that referenced
this pull request
Jul 23, 2026
…1.2.0) Restacked onto ben/items-ir (#144) with the standalone fixes split to #143; the pre-restack 43-commit history is preserved at ben/compaction-pre-split. The Compaction item + the conversation.compact step type (kind v1.2.0): a context-compaction boundary recorded as its own step between the turns it separates, with trigger, summary, pre_tokens, and the anchor-based kept contract (kept_from + expand_kept; the wire carries the expanded contiguous run). Compaction provenance is a closed typed set — no catch-all extra; native detail richer than the typed fields is deliberate loss. Per-provider support, each verified against the real harness TUI by resuming projected sessions (versions in docs/agents/tui-drift-check.md): - claude: compact_boundary + synthetic summary → typed boundary and back, preservedMessages as the kept run, transcript-only + isMeta render flags, empty-turn thinking-block projection. - codex: compacted marker → wholesale boundary (2026-07 payload: encrypted summary + prefix-keep documented as deliberate loss); projector emits the event_msg/context_compacted marker the TUI renders, with native turn_context placement. - opencode: per-boundary summary pairing, compaction-host parent redirects, tailStartID → kept anchor. - pi: compaction entries with parent-chain kept expansion; model_change/thinking_level_change/label as typed events with the chain's model context threaded into projected assistant messages. - copilot: the observed 1.0.68 session.compaction_start/complete pair mapped both directions (wholesale + summary). - gemini/cursor: documented no-boundary. Verification: executable round-trip oracle (testing::{check_view_invariants, assert_fixpoint}), compaction proptests, real captured compacted sessions for claude/codex/pi/ copilot/opencode as fixtures with looped stability tests, the cross-harness compaction matrix (kept-anchor capability flags), the env-gated real-session matrix mode, and import-time event-type surfacing.
|
🔍 Preview deployed: https://5482a221.toolpath.pages.dev |
Extracted from #108 as its structural prerequisite — the items IR without any compaction content. - ConversationView.turns/.events (two parallel vecs) become one ordered items: Vec<Item> (Turn | Event), preserving the source stream's exact interleaving. turns()/events() stay as iterators. - derive_path emits conversation.event steps for events (previously dropped — a Claude session lost its attachments and system entries on import/export), resolves duplicate step ids by content-aware renaming instead of erroring, and splices events onto the head's ancestry so they don't dangle as false dead ends. extract_conversation restores items, spliced parents, session files_changed, and synthetic-model actors. - Every provider's to_view now builds the ordered item stream. Native compaction markers stay opaque (a generic event, or pi's System turn) — typed boundaries are the follow-up's concern. - The claude projector emits events inline at their item position; real Claude interleaves attachments and system entries with turns, and the trailing-pass writer regrouped them at the end of the file. wire_order_roundtrip pins the projected entry-type sequence to the captured fixture's, and the caveat entry keeps isMeta. - Property tests: unique derived step ids, derive → extract → derive stability, and byte-identical replay dropping, over randomized turn/event interleavings with id collisions, dangling parents, and file mutations.
path-cli and toolpath-cli 0.16.0 collided with the 0.16.0 already released from main on 2026-07-16 — the identical version edit merged cleanly, so the collision was invisible; both move to 0.17.0. toolpath, toolpath-git, toolpath-github, toolpath-dot, and toolpath-md had no source changes on this branch; their bumps are reverted so the release does not republish five unchanged crates. site/_data/crates.json brought in line with the surviving bumps.
to_view assembled items as all surviving turns followed by all events, so the derived path misordered every conversation.event step — a compacted marker landed after the post-compaction turns, losing the stream position that gives the boundary its meaning. Events now carry a turn watermark recorded at push time and assembly merges the two streams on it, reproducing the rollout's exact interleaving. The compaction_roundtrip suite pins the full item shape through derive -> extract, replacing the now-false 'the IR drops it on the floor' known-limitation note. Also drops the dead CompactedItem type (defined and re-exported, never used - it belongs to the compaction-provenance follow-up) and the stale projector comment describing compaction handling that does not exist on this branch.
A byte-identical re-emission of an already-derived turn (the Claude chain-merge replay shape) was compared against the stored step's resolved bytes. When the original had been spliced onto an intervening event, or renamed after an id collision, the resolved forms no longer matched, so the replay was renamed-kept, became the head, and orphaned the original and its event as false dead ends. Resolved forms are not comparable across the stream - splicing rewires parents and parent mappings mutate as colliding steps rename - so replays are now recognized before resolution, by comparing source bytes per source id (seen_turn_sources / seen_event_sources). push_step keeps its rename machinery for genuine same-id-different-content collisions. Event steps now also stamp source_parent (null = positional chaining, otherwise the resolved parent), and extract restores it. Without the stamp, extract handed back the resolved chain, erasing the source distinction between an event that named a parent and one chained by position - two such events with equal data collapsed into one on the next derive, breaking gen-2 stability. This also fixes the review finding that an event natively parented across an adjacent event lost that linkage on project-out. Restores the two duplicate-id tests deleted from the merge base (the renamed-parent-reference pin and the Claude reused-attachment-uuid case), adds a regression test for the replay-after-splice shape, widens the replay property to spliced originals, makes duplicate event ids and turn/event id collisions reachable in the proptest generator, and drops the stale proptest-regressions seeds carried over from the compaction branch (they no longer match this generator's shape). Also removes the always-equal prev_anchor_step threading (compaction scaffolding that made the second splice disjunct unreachable).
… ref The regenerated pi fixture was a degenerate capture - one assistant turn, zero tool calls - so the pi real-fixture roundtrip and the cross-harness matrix tool/usage invariants passed vacuously. The previous capture (11 assistant turns, 10 tool calls, 10 tool results) is restored until a fresh elicitation produces a session with real tool activity. ConversationView.items doc no longer mentions compaction boundaries or a compactions() accessor - both belong to the follow-up PR.
The seen_uuids replay strip was untested drop logic whose comment cited a function that does not exist on this branch. The comment now describes the observed re-emission pattern honestly (the format docs do not cover it), and two unit tests pin the behavior: byte-identical replays are stripped with the tool result assembled once, and compact boundaries are exempt so repeated markers survive to the item stream. compaction_roundtrip's header claimed the boundary is dropped on the floor - it survives as an inline event now - and a new test pins its stream position through derive -> extract. wire_order_roundtrip's module comment now states exactly what is pinned (the entry-type sequence of the direct pipeline) and what is not (parentUuid values, 11/45 of which legitimately diverge for re-synthesized tool-result carriers).
…attachment-only user turns opencode: empty-text user messages are now suppressed only when they hosted a compaction part - an attachment-only user message emits a turn again, as before the items migration. The msg_redirects/resolve_parent docs describe the actual implementation (the old comments described a removed boundary-index design), the always-equal last_anchor_id duplicate of last_turn_id is gone, and the projector's never-read id_map scaffolding is removed. Restored compaction_becomes_event and fixture_loads_with_compaction_part, extended them to pin the boundary event's in-stream position and parent, added a derive -> extract survival test, and covered monotonize_times and the redirect path with unit tests. pi: restored test_compaction_produces_system_turn (deleted with no replacement) and added resolve_item_parent tests for the nearest-ancestor walk, the virtual root, and the cycle guard.
… event positions Cursor was the only provider discarding its native compaction marker: capabilityType-22 bubbles were skipped outright, leaving the typed- compaction follow-up nothing to anchor on and dropping the marker from projected sessions. The bubble now becomes a summarization event at its stream position, and the projector writes it back as a well-formed marker bubble (shape verified against real state.vscdb rows), so the marker survives a full cursor -> toolpath -> cursor round-trip. The real-DB sanity test now checks event invariants, and docs/agents/formats/cursor.md documents the marker so the code's doc references resolve. copilot: the watermark merge had no positional coverage; four unit tests pin where events land relative to in-progress, dropped, leading, and trailing turns.
…s, cargo fmt The version bumps shipped without their required CHANGELOG.md and site/_data/crates.json updates; both are now in place (crates.json in the earlier renumber commit). CLAUDE.md per-crate test counts match the suites as of this branch. Remaining files are cargo fmt output - the branch previously failed the format gate with diffs in five crates.
…curacy Round-2 adversarial review of the fix stack surfaced two defects in the replay skip and a handful of accuracy residues. derive_path: turn replays are now identified in a prepass and excluded from every per-turn structure. The in-loop skip still consumed a turn_groups slot and a synthesized-id slot, so a byte-identical same-group replay at a group tail silently dropped the group's token_usage stamp, and an id-less turn after a skipped replay shifted its step-NNNN id - both violating the replay-is-a-no-op contract. Unreachable through shipped providers (only claude sets group_id, and it strips replayed uuids before derive), but real contract defects; both now have regression tests. opencode: the user-turn parent synthesis (the one round-1 test gap the fix round missed) is pinned; the format doc's event types match the code (part.compaction / part.retry); the projector comment no longer claims compaction boundaries land at position (events are dropped on projection); a dangling see-module-note reference is gone. claude: the dedup comment is reframed as defensive - no 2.1.x session on this machine shows the re-emitted block, so the code guards the shape rather than documenting an observed one; the wire_order comment attributes the 11 parentUuid divergences by projected outcome instead of a source-parent breakdown that miscounted. codex/convo: two comments naming nonexistent compaction machinery reworded.
ben-emp
pushed a commit
that referenced
this pull request
Jul 27, 2026
…1.2.0) Restacked onto ben/items-ir (#144) with the standalone fixes split to ben/compaction-pre-split. The Compaction item + the conversation.compact step type (kind v1.2.0): a context-compaction boundary recorded as its own step between the turns it separates, with trigger, summary, pre_tokens, and the anchor-based kept contract (kept_from + expand_kept; the wire carries the expanded contiguous run). Compaction provenance is a closed typed set — no catch-all extra; native detail richer than the typed fields is deliberate loss. Per-provider support, each verified against the real harness TUI by resuming projected sessions (versions in docs/agents/tui-drift-check.md): - claude: compact_boundary + synthetic summary → typed boundary and back, preservedMessages as the kept run, transcript-only + isMeta render flags, empty-turn thinking-block projection. - codex: compacted marker → wholesale boundary (2026-07 payload: encrypted summary + prefix-keep documented as deliberate loss); projector emits the event_msg/context_compacted marker the TUI renders, with native turn_context placement. - opencode: per-boundary summary pairing, compaction-host parent redirects, tailStartID → kept anchor. - pi: compaction entries with parent-chain kept expansion; model_change/thinking_level_change/label as typed events with the chain's model context threaded into projected assistant messages. - copilot: the observed 1.0.68 session.compaction_start/complete pair mapped both directions (wholesale + summary). - gemini/cursor: documented no-boundary. Verification: executable round-trip oracle (testing::{check_view_invariants, assert_fixpoint}), compaction proptests, real captured compacted sessions for claude/codex/pi/ copilot/opencode as fixtures with looped stability tests, the cross-harness compaction matrix (kept-anchor capability flags), the env-gated real-session matrix mode, and import-time event-type surfacing.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Extracted from #108 as its structural prerequisite — the items IR without any compaction content. #108 will restack on this branch.
ConversationView.turns/.events(two parallel vecs) become one ordereditems: Vec<Item>(Turn | Event), preserving the source stream's exact interleaving;turns()/events()remain as iterators.derive_pathemitsconversation.eventsteps for events (previously dropped — a Claude session lost its attachments and system entries on import/export), resolves duplicate step ids by content-aware renaming instead of erroring, and splices events onto the head's ancestry so they don't dangle as false dead ends.extract_conversationrestores items, spliced parents, sessionfiles_changed, and synthetic-model actors.to_viewbuilds the ordered item stream. Native compaction markers stay opaque here (a generic event, or pi's System turn) — typed boundaries are the follow-up's concern.wire_order_roundtrippins the projected entry-type sequence to the captured fixture's, and the caveat entry keepsisMeta.All 7 quality gates pass. Verified across every provider crate's full test suite.
Need help on this PR? Tag
@codesmith-botwith what you need. Autofix is disabled.