Local-first diary: shared Rust core, offline SSR, and direct Surreal sync - #10
Conversation
|
9242e2a answers the jank feedback: the transcript is now genuinely optimistic.
A second adversarial review pass (20 agents) over this change confirmed four defects, all fixed and re-verified in-browser: a mid-flush snapshot could briefly blank a saved-but-unreported bubble (new provisional bucket), a page opened mid-flush could draw an entry twice when the late report arrived (server-rendered ids retire bubbles without redrawing), the offline form-POST fallback could let the worker answer a failed POST with the cached page — silently eating text (non-GET navigations now surface the error, and the page skips the fallback when provably offline), and the "No messages yet" placeholder stayed visible above the first delivered message. 🤖 Generated with Claude Code |
Stage 1 of loader/clientLoader-style isomorphism for topcoat + SurrealDB: - crates/diary-core: the whole queue, written once against Surreal<Any>. contract = the replay protocol (wire shape, window, normalization, response classification); outbox = the queue itself. Tested natively against mem://; the phone runs the identical code against indxdb:// (SurrealDB's IndexedDB engine). - crates/diary-worker: wasm-bindgen skin + the browser fetch transport, in its own workspace because it carries a [patch.crates-io] on surrealdb-core (kvs/ds.rs and the TIMEOUT operator still call tokio::time::Instant::now(), which panics on wasm — upstream #6711's unresolved half). The server workspace always builds pristine code; just wasm materializes the sha256-verified tarball + committed patch. - src/app/diary.rs now consumes diary_core::contract — one definition of the protocol compiled into both binaries. - src/app/diary_sync.rs serves wasm-dist/ at three stable routes with hash-paired versioning (loader no-cache; glue/wasm immutable only under the current ?v=), read from disk so dev and check need no wasm toolchain. - sw.js/diary.js reduce to browser glue (caching, Web Lock, Background Sync, BroadcastChannel, DOM); the old IndexedDB queue is drained by a one-way migration under the flush lock. Verified end-to-end in Chrome: enqueue (shared normalization), reload persistence, flush against the live endpoint (401 → blocked "auth", entry retained), import idempotency, and the real service worker (importScripts pair, in-worker wasm init, flush, broadcast). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013wywsqAm7q6geoGnH3GRDg
An 18-agent review (six lenses, each finding independently re-verified) ran over the branch; everything that survived is fixed here: - Over-length entries QUEUE again instead of bouncing into the lossy form-POST fallback: enqueue now refuses only empty text (contract::normalize_lines), so the server's 422 marks the entry failed with its text preserved — the old protocol, and the "queued diary text is never silently dropped" invariant. Offline, the old behavior lost the text outright. - The service worker primes its own glue/wasm pair at activation (pointees before the loader pointer, version-checked), so an update without a /diary page visit can still flush offline; only a fully primed pair licenses deleting the previous one. - primeCaches follows the worker's immutable-only rule for versioned sync bytes — a deploy-race answer under a stale ?v can no longer be pinned by the page half. - diary_sync::dist() re-stats after reading so a live `just wasm` can't mint an immutable ?v for a torn glue/wasm combination. - vendor-surrealdb-core.sh downloads via a temp file and deletes a checksum-failing tarball instead of wedging forever. - .dockerignore excludes crates/diary-worker/target and vendor/ so local image builds can't reuse host-built artifacts. - Comments/docs no longer claim `just check` covers diary-worker (it is excluded; breakage surfaces at `just wasm` / the Docker wasm stage) and the stale-pair cache weight is stated as ~18 MB decoded, not ~4 MB. Verified in Chrome: a 70,000-char entry queues pending, empty text still refuses, and a freshly activated worker holds its loader+glue+wasm trio in Cache Storage with no page-side priming. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013wywsqAm7q6geoGnH3GRDg
A sent message now renders synchronously in the submit handler and never leaves the screen: it moves through optimistic -> queued -> delivered buckets reconciled by qid, and flips to a permalink-bearing message matching the server's own markup when the flush report lands. The old post-flush window.location.assign (a guaranteed disappear-and-reappear) is gone, pinned against regression by a needle test; the worker now refreshes the cached /diary copy itself after a saving flush. To make that possible the flush report says WHAT saved, not just how many: SendOutcome::Saved carries the server-assigned SavedRef (collision probes can bump the second, so the response's id/written_at are the truth) and FlushReport gains saved_entries (qid + id + written_at + body); diary_enqueue returns the new qid so drafts reconcile exactly. A 20-agent adversarial review confirmed four defects, all fixed and browser-verified: a mid-flush snapshot could blank a saved-but-unreported bubble (new provisional bucket keeps it); a page opened mid-flush could draw an entry twice when the late report arrived (server-rendered ids retire bubbles without redrawing); the offline form-POST fallback let the service worker answer a failed POST with the cached page, silently eating the text (non-GET navigations now surface the error, and the page skips the fallback entirely when provably offline); and the "No messages yet" placeholder stayed visible above the first delivered message. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013wywsqAm7q6geoGnH3GRDg
Every online send still flashed: the bubble spent the flush round-trip
as a dashed diary-message-queued article with a longer meta line
("Aug 2, 2:15 PM · queued — will sync") before snapping to the delivered
form. Transitional bubbles (drafts, store-queued, mid-flush provisional)
now render exactly like a delivered message — same bubble style, same
stamp shape from the local clock, stamp colored like quiet-link — so
delivery only swaps a span for an identically-styled anchor and rewrites
nothing visible (verified with a MutationObserver timeline: the meta
text is byte-identical across the transition).
Status is news-driven instead of default: the dashed styling and
"queued — will sync" label appear only after a flush report says the
queue is actually blocked (offline / signed out), and "failed" only when
the server rejected the entry — the same shape messaging apps use.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013wywsqAm7q6geoGnH3GRDg
Desktop keyboards send on Enter like any chat box (Shift+Enter keeps the newline, an IME composition's Enter never fires a send); touch keyboards keep Enter as newline and the send button does the sending — gated on the (hover: hover) and (pointer: fine) media query. The textarea autofocuses via the HTML attribute plus a JS fallback for the navigations Chrome skips it on, same gate so phones don't pop the keyboard, and a send keeps the caret in the box. Testing this surfaced a missed call site in the vendored wasm time patch: kvs/tasklease.rs still used tokio::time::sleep, whose internal Instant::now() panicked and killed every spawned background task (index compaction, event processing, tombstone reclaim) with a console exception per task on each page load. The patch now swaps it onto wasmtimer like the other two files; a fresh page load runs exception- free. (dbs/executor.rs has two more tokio::time::timeout calls, but both sit behind a transaction_timeout this build never configures — left for the upstream PR, noted in docs/diary-sync.md.) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013wywsqAm7q6geoGnH3GRDg
The release that matters for this branch: 0.5.0 put topcoat-router's hyper/tokio behind an opt-in serve feature, which is what diary-sync.md roadmap item 3 (SSR in the service worker) named as the upstream ask. The upgrade itself is small. Error constructors moved to topcoat::router::error (imports in 14 files; the query_params error= attribute resolves the name itself), session Config became SessionConfig, and the CLI pin in deploy/Dockerfile follows the crate. The other breaking changes are no-ops here: no layouts (#166), no AssetConfig::hosted_at, and #179's boolean-attribute rework leaves static attr="" spellings untouched — only the initially-true :hidden bindings on the planes page serialize as hidden="" where 0.4.0 wrote hidden="true", identical to a browser (diffed against prod). Verified beyond just check: asset bundle with CLI 0.5.0, a booted server (routes, diary gate redirect, em-dash layer, typed byte routes), and a real browser pass over the planes page — signal toggles, :hidden/:class/ :aria-selected bindings, @click, shard round-trips, zero console errors. docs/topcoat-notes.md is rewritten for 0.5.0 (release tags are plain v0.5.0 again; new wasm section), and roadmap item 3 drops from "blocked upstream" to one wall, probe-confirmed: page/component render futures are + Send with no wasm cfg (topcoat-view Component::render and router PageRenderFn). spawn_local + a oneshot channel works around it; the clean fix is a small upstream PR cfg-gating both bounds. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q5fZ3v3NUqosEBFmKXhbjn
Phase 0 of the local-first diary plan: pure refactor, no behavior change. - eastern.rs moves to diary_core::eastern (with its D1 corpus fixture and the capture.sh/README pointers): diary entry ids share the projection and the wasm worker must compute them too. The lifting archive re-exports it, and podrick's #[path] re-mount becomes a normal dependency. - New diary_core::store: DiaryEntry, entry_key, entry_page/entry_by_id/ insert_entry/remove_entry, and save_queued_entry -> save_entry, all generalized from &Data to the crate's one Surreal<Any> handle. diary.rs keeps thin adapters so every call site and outage branch is unchanged. - The probe-and-dedupe algorithm gains its first native tests (mem://): replay dedupe, different-body probing, the killer bumped-replay walk, probe exhaustion, page reads. The mem:// helper defines the table shape because a fresh 3.2 store errors on undefined tables. - diary-core gains jiff with a deliberately minimal feature set (std + tzdb-bundle-always + perf-inline, no js) - verified compiling for wasm32 via just wasm. - build.rs reruns the Tailwind scan when crates/diary-core/src changes, ahead of shared views living there. just check green (425 tests), just wasm green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q5fZ3v3NUqosEBFmKXhbjn
Phase 1 of the local-first diary plan. - crates/diary-core/outbox.rs rewritten around a local diary_entries table: server fields plus state (synced/pending/failed), reason, enqueued_at. A queued entry is a pending row; delivery flips state IN PLACE (the delete-before-report gap that forced the page's provisional-bubble machine is gone); a server bump re-keys to the server identity in one transaction, releasing the row instead when a different pending neighbor holds that key. Ids are predicted at enqueue with the same probe-and-dedupe loop the server runs, so a double-tap converges locally instead of minting a twin the server would store twice. - The v1 diary_outbox drains into the single store inside open() as a STANDING step (deploy skew re-creates it); unprojectable timestamps port under synthetic failed-* keys - never dropped. The legacy IndexedDB import targets the new table with the same idempotency. - diary.js rewritten around a server-shipped <template id="diary-bubble"> and data-id reconciliation: clone synchronously on submit (bubble before any await), fill the predicted id when enqueue resolves, toggle states from reports, one renderFromStore path for labels/prunes/mid-flush opens. The five-bucket painter, provisional Map, and hand-built article builders are deleted; every bubble class now lives in .rs markup where the Tailwind scan sees it (previously the discard button's utilities survived only because unrelated pages used them). 585 -> 476 lines. - diary_enqueue returns the placed row as JSON; diary_snapshot returns only non-synced rows; diary_discard spares synced history. - Tripwire tests re-pinned same-commit; new native coverage: same-second dedupe, forward probing, in-place state flips, bump re-key, the bump-vs-pending-neighbor walk, standing-drain idempotence, synthetic keys. docs/diary-sync.md's now-wrong sections corrected (full rewrite stays scheduled for the cleanup phase). Verified: just check green (431 tests), just wasm, and a real-Chrome pass against the dev stack - synchronous first paint, predicted id == server id with the permalink live from birth, reload dedupe by data-id, offline enqueue -> "queued - will sync" -> Background Sync auto-delivery after the server returned, double-tap converging to one entry, 422 -> failed bubble with reason + working discard, zero console errors. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q5fZ3v3NUqosEBFmKXhbjn
Phase 2 of the local-first diary plan. - contract: SNAPSHOT_PATH, the strict SnapshotWire/SnapshotEntry shapes (deny_unknown_fields like the push side), and classify_pull with the same three-way rule as pushes - a 200 that is not our exact JSON is a captive portal, never an empty diary. That rule is load-bearing: reading hotel-wifi HTML as "zero entries" would delete every synced row. - diary_core::sync: the two-method Remote trait (deliberately NO Send bounds, mirroring outbox::flush), sync::run (flush then pull, pull skipped on auth-block), and apply_pull - the diff computed in Rust, applied as one small transaction (creates as synced / updates and deletes guarded on state = 'synced'; pending and failed rows are never touched, even at a matching id), wrapped in the Resource-busy conflict retry. FlushReport gains `pulled` (stale pages ignore it). - The native suite grows the plan's walks: garbage-pull no-op, the adversarial apply (server rows at pending/failed ids), auth skips pull, and the two-device same-second convergence test running against a REAL second mem:// store whose push IS store::save_entry - byte-for-byte the shape the direct client-to-SurrealDB transport takes in the flag phase. - Server: GET /api/diary/snapshot (admin cookie, no-store, explicit projection); unlisted/untrackable tests extended. - Worker: diary_flush becomes diary_sync(push_url, pull_url) over an HttpRemote; sw.js runs the whole pass inside its one Web Lock hold, so a pull's snapshot is always newer than the last save. - docs/auth.md PWA section: the offline surface is now the FULL mirror under the same device-possession trust model, stated consciously. Verified: just check green (435 tests), just wasm, and in Chrome against the dev stack - first sync hydrated exactly the 10 rows the device did not already hold, steady-state pulls report 0, and a server-side delete propagated as pulled: 1 on the next pass. Zero console errors. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q5fZ3v3NUqosEBFmKXhbjn
Phase 3 of the local-first diary plan - one renderer, two hosts.
- diary_core::views (feature "view"; topcoat default-features=false): the
ONE definition of transcript, bubble (all states, every part present and
hidden-toggled so diary.js moves a bubble between states without
rebuilding it), compose form, the page-JS <template>, and the minimal
offline chrome. Components are deliberately PURE - zero awaits - which
satisfies Component::render's unconditional +Send bound on wasm for
free. Stamp/url formatting moved here with its tests.
- src/app/diary.rs renders through the shared components inside the site
shell (server history articles now carry the full toggleable bubble
shape - a deliberate markup delta); the template IS the bubble
component now.
- crates/diary-worker grows the serve-less router (0.5.0 feature split:
router+view+discover, no hyper anywhere): #[page("/diary")] (clamped
?page=, never redirects) and #[page("/diary/{path}")] over the mirror,
reads bounced through spawn_local + oneshot (the Send bridge),
Router::handle dispatching in a build-once leaked router; new export
diary_render(url, assets). +0.6 MB on the 18 MB module.
- /diary-sync.js loader now carries the hashed stylesheet + diary.js URLs
(resolved server-side via AssetConfig::resolve - no Asset machinery on
wasm); a CSS-only deploy thereby rolls a new worker version, and the
worker primes those assets on activate before dropping stale ones.
- sw.js: navigations stay network-first; the failure path renders from
the mirror instead of serving a stale cached copy - diary-page-v1 and
refreshPageCache are gone (activate auto-deletes the old cache), the
stub remains as the wasm-refused last resort, failed POSTs still throw.
- Two duplicate-asset-route panics found at router boot (asset! twice for
diary.js; stylesheet!() twice) - fixed by single shared consts
(chrome::SITE_CSS, diary::DIARY_JS) and recorded in the crib sheet with
the #[component]-cx-naming gotcha.
Verified: just check green (436 tests), just wasm, just build, and in
Chrome with the server process killed - the worker rendered /diary
complete and styled from the mirror (13 articles incl. an offline write
as a pending row with the will-sync label), the offline permalink page
rendered with stamp/body/delete, and after the server returned the flush
report flipped the WORKER-rendered pending article to synced live: the
page JS cannot tell which renderer drew the HTML. Zero console errors.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q5fZ3v3NUqosEBFmKXhbjn
Phase 4 of the local-first diary plan. Flag-off (the default) changes nothing; flag-on removes the app server from the sync path entirely - its one remaining role is identity. - schema.surql: diary_entries grants select+create WHERE $access = 'diary_sync' (inert until the access method exists); every other table stays PERMISSIONS NONE. The access method itself is CONFIG, not schema: src/data.rs defines it at bootstrap from DIARY_SYNC_JWT_PUBLIC_KEY and REMOVEs it when unset - flag-off means surface-off. - POST /api/diary/token (admin cookie + same-origin; 404 while off) mints 15-minute ES256 record-access tokens via jsonwebtoken. The claims carry `id` (diary_device:admin) - that is what makes SurrealDB build a fully permission-bound RECORD session. - diary_core::sync::DirectRemote: the remote IS a Surreal<Any> handle; push runs the same store::save_entry probe the replay endpoint runs, pull the same snapshot read - one algorithm, two engines, tested store-to-store natively. New wipe guard in apply_pull: an EMPTY snapshot never deletes a populated mirror. - Worker: diary_sync gains the direct endpoint arg (loader-advertised via DIARY_DIRECT_SYNC_ENDPOINT); each pass mints a token, opens a fresh short-lived WEBSOCKET, authenticates, and verifies the $access canary; any arming failure falls back to the HTTP endpoints for that pass. Probes on the pinned 3.2.3 container reshaped the design and are now load-bearing documentation (docs/diary-sync.md "Direct sync"): - TYPE JWT ON DATABASE grants a Viewer-role floor that reads EVERY table regardless of PERMISSIONS; the access MUST be TYPE RECORD WITH JWT. - The stateless-http engine's authenticate() does not stick on server 3.2.3 (each later request arrives anonymous), so the endpoint MUST be ws/wss - and SurrealDB filters denied reads to EMPTY rather than erroring, which is why the canary + wipe guard exist: a silently deauthed session must never read as "an empty diary". - jsonwebtoken needs PKCS#8 PEM. SurrealDB's own CORS is permissive, so prod is a plain flag-gated db. subdomain through the tunnel (cloudflare-deploy.md + railway-deploy.md runbooks added). Verified: just check green (439 tests incl. DirectRemote store-to-store, the wipe-guard walk, auth classification), just wasm (19.1 MB raw / 5.1 MB gz, +250 KB for protocol-ws), and end to end in Chrome against a flag-on instance: the mirror hydrated over the websocket, and a new entry landed in SurrealDB under its predicted id with exactly ONE app server request in the log - the token mint. No replay POST, no snapshot GET. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q5fZ3v3NUqosEBFmKXhbjn
Phase 5 of the local-first diary plan - docs and truth reconciliation. - docs/diary-sync.md rewritten as the architecture doc: the isomorphism is built, the Shape section describes the six shared modules and the worker binary as they exist, and the roadmap section became "What remains (all optional)" - wasm-opt, the upstream Send-bound PR, a future changefeed pull, live queries. - docs/auth.md PWA section: offline reads are worker SSR from the mirror (the cached-HTML copy is gone); sync bullets cover the flagged direct path; device-possession trust model restated. - CLAUDE.md diary bullets rewritten for the single store, one-definition markup rule, and the duplicate-asset!-route panic gotcha. - Project memory updated: the isomorphism memory records the five phase commits, the design keystones, and the probe findings now serving as load-bearing documentation. just check green (439 tests). diary.js stands at 466 lines (from 585), with the five-bucket painter, provisional map, and hand-built DOM articles gone - what remains is worker lifecycle, cache priming, the template-clone bubble glue, and Enter-to-send. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q5fZ3v3NUqosEBFmKXhbjn
36daa5a to
ded32b8
Compare
Local-first diary foundation
The diary remains a local-first offline PWA written in Rust on both sides of the wire. This revision also makes the entry lifecycle canonical so later business changes do not have to be threaded separately through the server, queue, sync engine, and wasm worker.
Architecture
EntryContentis the one canonical home for business fields and replay equality.ComposedEntry,DiaryEntry, andSavedRefrepresent composition, placement, persistence, and acknowledgement without transport-specific copies.ComposeCommandvalue into wasm, so new entry content does not expand the wasm export signature.CURRENT_SCHEMA_EPOCHis an exact deployment fence. There are no historical wire DTOs, compatibility adapter ladders, or per-entry version fields; stale clients retain their outbox and retry after updating.diary-storeWeb Lock, and cached wasm/server handles recheck their migration ledger before use.Intended future change surface
A durable business field should normally require changes only in:
EntryContentand any intrinsic validation;The sync sequencer, worker exports, service worker, collision placement, dedupe, queue state transitions, reconciliation, flush reports, and acknowledgements should not need field-specific mapping.
Rollout and direct-sync safety
crates/diary-workerremains its own workspace so the wasm-only SurrealDB patch cannot affect the server.Verified
just checkjust wasmjust buildUpstream constraint
SurrealDB core 3.2.3 still needs the committed wasm time patch. It is applied only inside the worker workspace through
[patch.crates-io]andscripts/vendor-surrealdb-core.sh.