Ledger engine: crash-safe, resumable drill_events migration — no Redis - #7

Open

ar2rsawseen wants to merge 42 commits into

mainfrom

poc/ledger-no-redis

Commits on Aug 17, 2026

feat: ledger engine — chunk checklist + per-chunk staging, no Redis (A/B vs classic)

ar2rsawseen
and
claude
committed
4e58e48
bench: kill-drill supports both engines via DRILL_ENGINE

ar2rsawseen
and
claude
committed
c0b71cf
feat: Countly-branded live dashboard for the ledger engine (/viz)

ar2rsawseen
and
claude
committed
d2ae949
feat: complete the ledger engine — DLQ, bisection, breaker, backpressure, monitor, dry-run, report

ar2rsawseen
and
claude
committed
7f85b6c
refactor: single-architecture service — remove the legacy engine entirely

ar2rsawseen
and
claude
committed
7845383
fix: honest completion + operator recovery (found by live failure drill)

ar2rsawseen
and
claude
committed
1505849
feat: ingestion-matching transform — adopt shared spec + differential harness (D4, #6)

ar2rsawseen
and
claude
committed
0d23752
fix: per-process dedup-canary table name (multi-pod create/drop race)

ar2rsawseen
and
claude
committed
691609a
docs: operational runbook (flow, incident responses, in-place variant)

ar2rsawseen
and
claude
committed
fa3d1a2
feat: poison-pill quarantine — auto-split chunks that crash the process

ar2rsawseen
and
claude
committed
ed9ef40
feat: DLQ waive — the terminal operator decision for unmigratable docs

ar2rsawseen
and
claude
committed
d0ac2c8
feat(viz): full operator console — controls, DLQ panel, coercion report

ar2rsawseen
and
claude
committed
57eb793
feat(viz): self-service migration console — guided runbook, preflight, verification

ar2rsawseen
and
claude
committed
4d6f710
feat: pod scaling UX, real preflight actions, chaos-hardening

ar2rsawseen
and
claude
committed
63413e9

Commits on Aug 18, 2026

feat(viz): configuration card — current knobs, defaults, when-to-change, state location

ar2rsawseen
and
claude
committed
58baa64
fix: null-cd sweep ordering — close a silent data-miss window (found by review)

ar2rsawseen
and
claude
committed
87c5478
docs+setup: fresh-operator path — compose/.env rewritten, README split, null-cd preflight count

ar2rsawseen
and
claude
committed
c9d5620
feat(viz): serve the console at / — /viz kept as alias

ar2rsawseen
and
claude
committed
b4da411
fix(engine): a startup failure no longer kills the console

ar2rsawseen
and
claude
committed
7ebcd8c
feat: ledger rebuild from data + per-collection scoping of all live cd-window queries

ar2rsawseen
and
claude
committed
9cf0073
feat(k8s): ready-to-apply Kubernetes manifests

ar2rsawseen
and
claude
committed
d461d94
docs: decouple the transform spec from the unmerged platform PR

ar2rsawseen
and
claude
committed
0e2c34c
docs: reference countly-platform#1105 (surgical cd passthrough)

ar2rsawseen
and
claude
committed
82d4783
feat(verify): attribute duplicate ids to live ingestion vs migration

ar2rsawseen
and
claude
committed
900e016
feat: migrated provenance flag — 100% exact migrated/live distinction

ar2rsawseen
and
claude
committed
68cfe26
refactor: provenance without schema changes — (_id, cd) pairs replace the migrated column

ar2rsawseen
and
claude
committed
622dcf3
ci: run the full integration suite on every PR (MongoDB 7 + ClickHouse 26.4)

ar2rsawseen
and
claude
committed
081afc4
fix(viz): honest counters + DLQ fix-location guidance

ar2rsawseen
and
claude
committed
2243004
feat: scale-proof UI + self-driving preflight

ar2rsawseen
and
claude
committed
2575422
feat: billion-scale engine paths — async verify, partitioned dup scan, full DLQ drain

ar2rsawseen
and
claude
committed
06e4782
config: default SERVICE_NAME to drill-migrator — two required vars, not three

ar2rsawseen
and
claude
committed
43fa12d
test: migration under concurrent live ingestion

ar2rsawseen
and
claude
committed
930f9aa
feat: mass-DLQ safety — global pause guard, background replay, redo-then-replay dedup

ar2rsawseen
and
claude
committed
2f6d87d
feat(dlq): mass guard and panel report the disk cost

ar2rsawseen
and
claude
committed
c801daa
fix(viz): replay toast says started, not finished

ar2rsawseen
and
claude
committed
d7dbef0
feat: optional dashboard auth (DASHBOARD_PASSWORD)

ar2rsawseen
and
claude
committed
f2759d0
Revert dashboard auth — not needed

ar2rsawseen
and
claude
committed
c78e2df
docs: Docker multi-container scaling recipe

ar2rsawseen
and
claude
committed
8fa6c2a

Commits on Aug 20, 2026

feat: post-migration audits — source recount + sampled doc-per-doc comparison

ar2rsawseen
and
claude
committed
9652804
feat: per-commit source-count guard + cd-pruned id lookups

ar2rsawseen
and
claude
committed
363cc63
fix(audit): waived/pending DLQ docs explain their window's shortfall

ar2rsawseen
and
claude
committed
5e71394
test: de-flake live-parallel writer assertion

ar2rsawseen
and
claude
committed
a78df1a