Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
139 commits
Select commit Hold shift + click to select a range
ec84edb
Rebase on upstream hourly, add AI/LLM PR review
gburd Mar 10, 2026
568e532
ci: AI PR review — Open Code Review + Agora MCP history
gburd Jun 6, 2026
82cc694
ci: bump pg-history Bedrock read timeout past botocore's 60s default
gburd Jul 4, 2026
697193b
Add pg_fts contrib module: ftsdoc/ftsquery types and @@@ operator
gburd Jul 1, 2026
4ae7901
pg_fts: add pluggable analyzer reusing text-search configs (stage 2)
gburd Jul 1, 2026
23af4d3
pg_fts: add BM25 relevance scoring (stage 4)
gburd Jul 1, 2026
fc31f5c
pg_fts: add the bm25 inverted-index access method (stage 3)
gburd Jul 1, 2026
989585d
pg_fts: add selectable BM25 variants (stage 9)
gburd Jul 1, 2026
d7db020
pg_fts: add highlight and snippet auxiliary functions (stage 8)
gburd Jul 1, 2026
275ee30
pg_fts: add tsquery -> ftsquery migration converter (stage 11)
gburd Jul 1, 2026
4f0b162
pg_fts: add prefix queries term* (stage 6, part 1)
gburd Jul 1, 2026
4ec8b58
pg_fts: expose index-maintained corpus statistics for BM25 (stage 5)
gburd Jul 1, 2026
5b97874
pg_fts: document implemented stages and remaining roadmap (stage 15, …
gburd Jul 1, 2026
8e173bb
pg_fts: incremental maintenance via a pending list (stage 7)
gburd Jul 1, 2026
5a0249f
pg_fts: add phrase queries via positional ftsdoc (stage 6, part 2)
gburd Jul 1, 2026
816d537
pg_fts: document and test external-content indexing (stage 10)
gburd Jul 1, 2026
d2149a1
pg_fts: add fuzzy (term~k) and regex (/re/) queries (stages 13-14)
gburd Jul 1, 2026
d1bb5a3
pg_fts: add benchmark harness and finalize docs (stages 12, 15)
gburd Jul 1, 2026
5635df7
pg_fts: add BM25F multi-field weighting (future-work item)
gburd Jul 1, 2026
f4f456e
pg_fts: background merge of the pending list (future-work item)
gburd Jul 1, 2026
7d9868e
pg_fts: index-only BM25 top-k scoring with WAND bound (future-work item)
gburd Jul 1, 2026
5e5462d
pg_fts: trigram pre-filter for fuzzy matching (future-work item)
gburd Jul 1, 2026
ba511cc
pg_fts: update README for BM25F, merge, index-only top-k, trigram filter
gburd Jul 1, 2026
42eda04
pg_fts: make fts_search MVCC-correct (visibility filtering)
gburd Jul 1, 2026
06fc827
pg_fts: delta+varint posting compression (item 1)
gburd Jul 1, 2026
623d3f6
pg_fts: document-at-a-time WAND top-k with block-max impacts (item 2)
gburd Jul 1, 2026
6272d26
pg_fts: store per-document length in postings for exact BM25 length norm
gburd Jul 1, 2026
bba26f6
pg_fts: lazy posting reads + block-max page-cursor WAND (item b)
gburd Jul 1, 2026
2403dd3
pg_fts: add NEAR(a b, k) proximity queries
gburd Jul 1, 2026
59cbf12
pg_fts: FSM page recycling for merge-orphaned blocks
gburd Jul 1, 2026
d94b385
pg_fts: update README for per-doc-length, lazy WAND, NEAR, FSM recycling
gburd Jul 1, 2026
1ee32c5
pg_fts: amcanorderbyop ordering scan for ORDER BY d <=> query LIMIT k
gburd Jul 1, 2026
67e0479
pg_fts: vendor sparsemap v5.1.1, namespaced to __pg_bm25_*
gburd Jul 1, 2026
fd53ddd
pg_fts: on-disk trigram index for fuzzy/regex candidate narrowing (ta…
gburd Jul 1, 2026
867484c
pg_fts: README for amcanorderbyop, on-disk trigram index, vendored sp…
gburd Jul 1, 2026
efc7e29
pg_fts: AST-tiling trigram extraction for regex candidate narrowing
gburd Jul 1, 2026
2802e85
pg_fts: adaptive-k ordering scan (bounded work for small LIMIT)
gburd Jul 1, 2026
fb537cc
pg_fts: README - core roadmap complete, only refinements remain
gburd Jul 1, 2026
2072bf6
pg_fts: benchmark plan and harness for EC2 (us-east-2)
gburd Jul 1, 2026
788e61f
pg_fts: to_ftsquery(regconfig, text) normalizes query terms (EC2 faul…
gburd Jul 1, 2026
50025f3
pg_fts: invert trigram index to term-space (A) + fix sparsemap trunca…
gburd Jul 1, 2026
552f64b
pg_fts: document A+C trigram complement-encoding as future work
gburd Jul 1, 2026
680364b
pg_fts: design review + strategy re-plan from scale testing and algor…
gburd Jul 1, 2026
298399e
pg_fts: revise rebuild order - segment container first (dependency-co…
gburd Jul 1, 2026
b629682
pg_fts: segment-based storage architecture (rebuild step 1)
gburd Jul 1, 2026
105f007
pg_fts: size-tiered segment merge (rebuild step 2)
gburd Jul 1, 2026
a915029
pg_fts: checkpoint rebuild progress (steps 1-2 done); benchmark next
gburd Jul 1, 2026
11e0460
pg_fts: FOR-128 block posting codec + per-block max-impact (rebuild s…
gburd Jul 1, 2026
95d4572
pg_fts: precompute BM25 norm constants in WAND cursor (rebuild step 4a)
gburd Jul 1, 2026
b7dae19
pg_fts: sparse dictionary block index (rebuild step 5)
gburd Jul 1, 2026
e1fcbe5
pg_fts: block-max WAND pivot skip (rebuild step 7)
gburd Jul 1, 2026
3d73afd
pg_fts: FOR bit-pack intra-block posting payload (rebuild step 3b)
gburd Jul 1, 2026
ac4bb19
pg_fts: shared posting pages across terms (rebuild step 3c)
gburd Jul 1, 2026
4ba140e
pg_fts: restore lazy WAND paging over shared posting pages (fix 3c re…
gburd Jul 1, 2026
161f6b7
pg_fts: bench results (ranked top-k 18-22x faster than GIN) + WAND lo…
gburd Jul 1, 2026
25900dc
pg_fts: 10M-doc scaling results -- ranked top-k 25-38x faster than GIN
gburd Jul 1, 2026
3c74e2c
pg_fts: update STRATEGY_REPLAN with rebuild completion + benchmark ou…
gburd Jul 1, 2026
435f24d
pg_fts: Levenshtein-automaton fuzzy matching over the sorted dictionary
gburd Jul 2, 2026
f695485
pg_fts: MaxScore top-k for long queries, alongside BMW
gburd Jul 2, 2026
ceb722a
pg_fts: galloping intersection for skewed boolean AND
gburd Jul 2, 2026
566b53e
pg_fts: cross-version build guards (PG17/18/20) for AM routine + tabl…
gburd Jul 2, 2026
fb26005
pg_fts: exact boolean bitmap results skip the heap recheck
gburd Jul 2, 2026
10e74a2
pg_fts: single-term DFA fuzzy skips heap recheck (exact candidates)
gburd Jul 2, 2026
6a0ca1c
pg_fts: head-to-head vs ParadeDB pg_search 0.24.1 (PG17.5, 2M docs)
gburd Jul 2, 2026
fc711d9
pg_fts: record deferred-work completion + pg_search verdict in STRATE…
gburd Jul 2, 2026
0513d3d
pg_fts: index-only scan support -> heap-free counts (match pg_search)
gburd Jul 2, 2026
ae5c6ad
pg_fts: re-benchmark vs pg_search after index-only counts
gburd Jul 2, 2026
220ec70
pg_fts: start adaptive-k WAND at 128 to avoid recompute for page-size…
gburd Jul 2, 2026
932aa6e
pg_fts: keep adaptive-k at 64, grow x8 (best small-LIMIT ranked latency)
gburd Jul 2, 2026
f711fc9
pg_fts: tighter block-max WAND bound using per-block min |D|
gburd Jul 2, 2026
04b892b
pg_fts: revert adaptive-k growth to x4 (x8 over-grew and slowed large…
gburd Jul 2, 2026
1f3a5d0
pg_fts: finalize pg_search comparison (wins/ties 4 of 5 core queries)
gburd Jul 2, 2026
fb52ae1
pg_fts: block-skipping WAND seek (fix Q5/Q7 common-term ranked)
gburd Jul 2, 2026
87ad48a
pg_fts: DFA-guided dictionary skipping for fuzzy (fix Q6 fuzzy scan)
gburd Jul 2, 2026
ac13ee4
pg_fts: k-way merge the fuzzy union (fix the real Q6 cost)
gburd Jul 2, 2026
c98e1da
pg_fts: final pg_search comparison after Q5/Q7 + Q6 fixes
gburd Jul 2, 2026
475f41c
pg_fts: bound + tune adaptive-k ordering scan (Q5/Q7)
gburd Jul 2, 2026
ea57422
pg_fts: fts_count() -- MVCC-correct bulk count via the index (Q6 grou…
gburd Jul 2, 2026
1cb76b3
pg_fts: results after Q5/Q7 recompute fix + fts_count count path
gburd Jul 2, 2026
c3a9969
pg_fts: lazy per-column posting decode in the WAND cursor
gburd Jul 2, 2026
9f22f1a
pg_fts: bench lazy-decode WAND vs pg_search (Q5/Q7 12->9ms, gap 1.9x-…
gburd Jul 2, 2026
1986abf
pg_fts: implement bulkdelete tombstones (fix stale-posting correctnes…
gburd Jul 2, 2026
4c121c4
pg_fts: tsquery_to_ftsquery converts <-> faithfully to a phrase (was …
gburd Jul 2, 2026
7edb193
pg_fts: correct stale comments that no longer match the code
gburd Jul 2, 2026
b3ddeaa
pg_fts: fix stale block-encoding comment (FOR is implemented, not var…
gburd Jul 3, 2026
72ed532
pg_fts: multi-segment build flush to bound CREATE INDEX memory (featu…
gburd Jul 3, 2026
e4ecc10
pg_fts: size-tiered segment merge (feature 3/6)
gburd Jul 3, 2026
da5dad3
pg_fts: WAND-aware cost estimate for ranked ordering scans (feature 6…
gburd Jul 3, 2026
c846b7e
pg_fts: sublinear prefix lookup via the sparse block index (feature 5/6)
gburd Jul 3, 2026
7356134
pg_fts: widen segment-directory cap to 128 as a safe backstop (featur…
gburd Jul 3, 2026
71e4c7e
pg_fts: measure PFOR and decline it (feature 4/6 — evaluated, not war…
gburd Jul 3, 2026
2d93f60
pg_fts: tidy stale 'later tiered merge' comment (merge is implemented)
gburd Jul 3, 2026
47fbe87
pg_fts: fix two committer-review HIGH bugs (build term collision, can…
gburd Jul 3, 2026
714c334
pg_fts: address committer-review MED/LOW findings (hardening + commen…
gburd Jul 3, 2026
873da3f
pg_fts: update README to match the segmented architecture and current…
gburd Jul 3, 2026
cab148c
pg_fts: remove marketing language from comments and docs
gburd Jul 3, 2026
4d5b731
pg_fts: fix Windows/MSVC build of vendored sparsemap
gburd Jul 3, 2026
07759c4
pg_fts: make vendored sparsemap build on MSVC (Windows CI)
gburd Jul 3, 2026
b77eaaa
pg_fts: drop dead varint codec; guard sparsemap __builtin_expect for …
gburd Jul 3, 2026
ae67505
pg_fts: fix 32-bit crash in vendored sparsemap chunk-unset coalescing
gburd Jul 3, 2026
19fcae8
pg_fts: fix remaining 32-bit sparsemap truncations (complete the unse…
gburd Jul 3, 2026
79471cb
pg_fts: fix reused-heap-slot false negatives (per-segment tombstones)
gburd Jul 3, 2026
456a595
pg_fts: index oversized documents instead of rejecting them
gburd Jul 3, 2026
4b0c6d5
pg_fts: upgrade vendored sparsemap to v5.2.0 (+ 32-bit unset fix)
gburd Jul 3, 2026
6fae2fb
pg_fts: merge eagerly when oversized-doc inserts pile up segments
gburd Jul 3, 2026
7298b57
pg_fts: add real-corpus (2M Wikipedia) benchmark vs tsvector/GIN
gburd Jul 3, 2026
8ffbba0
pg_fts: upgrade vendored sparsemap to v5.2.1; add real-corpus pg_sear…
gburd Jul 4, 2026
35366c5
pg_fts: add CAPABILITIES.md (production-readiness / feature matrix + …
gburd Jul 4, 2026
9496af3
pg_fts: impact-ordered block skip directory for single-term ranked top-k
gburd Jul 4, 2026
e9a1811
pg_fts: sort impact directory by recomputed bound for exact early-stop
gburd Jul 4, 2026
a6e0a63
pg_fts: reduce top-k MVCC over-fetch from k*4 to k*2
gburd Jul 4, 2026
c6bbfc7
pg_fts: revert impact-ordered skip directory (measured: no pruning on…
gburd Jul 4, 2026
d399116
pg_fts: add isolation + TAP tests (concurrency, crash recovery, repli…
gburd Jul 4, 2026
7e21385
pg_fts: add SGML reference documentation; document tests + docs in RE…
gburd Jul 4, 2026
5c9fd6e
pg_fts: upgrade sparsemap to v5.3.0; use batched + cached contains APIs
gburd Jul 5, 2026
56df593
pg_fts: add the BM25L scoring variant (rank_bm25 parity)
gburd Jul 5, 2026
008057f
pg_fts: parallel index build (amcanbuildparallel)
gburd Jul 5, 2026
b8ee955
pg_fts: guard table_beginscan_parallel arity for PG17 (2 args) vs PG1…
gburd Jul 5, 2026
019bd1f
pg_fts: skip the forced merge-to-one after a parallel build
gburd Jul 5, 2026
38d3c0d
pg_fts: fts_merge() now compacts segments to one (explicit optimize)
gburd Jul 5, 2026
d23dcd0
pg_fts: parallel merge phase for merge-all (optimal index sooner)
gburd Jul 5, 2026
f13a546
pg_fts: track deferred enhancements in DEFERRED.md
gburd Jul 5, 2026
662be3d
pg_fts: compact the index after a parallel build (fix ranked-latency …
gburd Jul 5, 2026
1b4ab03
pg_fts: serial build also compacts to one segment (bm25_merge_all)
gburd Jul 5, 2026
20bed56
pg_fts: record parallel-build regression fix + ranked latency numbers
gburd Jul 5, 2026
9486f02
pg_fts: add same-hardware A/B confirming the ranked-latency regressio…
gburd Jul 5, 2026
5888cfd
pg_fts: 4-way competitive benchmark (pg_fts vs pg_search vs VectorCho…
gburd Jul 6, 2026
10b863d
pg_fts: make parallel merge/build writes actually concurrent (per-pag…
gburd Jul 6, 2026
5d2af7a
pg_fts: reclaim physical index bloat (low-page FSM bias + vacuum comp…
gburd Jul 6, 2026
34715c1
pg_fts: word-oriented FOR decode (5.7x faster posting unpack)
gburd Jul 6, 2026
7f4a05a
pg_fts: iterate the parallel merge to one segment (kill the serial me…
gburd Jul 6, 2026
8ce63c2
pg_fts: revert iterated parallel merge (measured worse at 2M)
gburd Jul 6, 2026
79d8dd6
pg_fts: benchmark parallel-merge concurrency, fts_vacuum reclaim, fas…
gburd Jul 6, 2026
232ff00
pg_fts: AIO-write + format-v3 investigation (profiled; codec rewrite …
gburd Jul 6, 2026
202f062
pg_fts: COUNT pushdown CustomScan (Option B, stage 1)
gburd Jul 6, 2026
1565432
pg_fts: parallel ranked CustomScan (Option B, stages 2+3)
gburd Jul 6, 2026
901e5dd
pg_fts: fts_vacuum reclaims bloat even when the index is already nseg=1
gburd Jul 6, 2026
46b2415
pg_fts: revert the parallel ranked CustomScan (no measured win); keep…
gburd Jul 6, 2026
33707d7
pg_fts: document fts_vacuum and the transparent COUNT pushdown (1.20)
gburd Jul 6, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 18 additions & 0 deletions .github/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
# Node modules
scripts/ai-review/node_modules/
# Note: package-lock.json should be committed for reproducible CI/CD builds

# Logs
scripts/ai-review/cost-log-*.json
scripts/ai-review/*.log

# OS files
.DS_Store
Thumbs.db

# Editor files
*.swp
*.swo
*~
.vscode/
.idea/
163 changes: 163 additions & 0 deletions .github/DEV_SETUP_FIX.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,163 @@
# Dev Setup Commit Fix - Summary

**Date:** 2026-03-10
**Issue:** Sync workflow was failing because "dev setup" commits were detected as pristine master violations

## Problem

The sync workflow was rejecting the "dev setup v19" commit (e5aa2da496c) because it modifies files outside `.github/`. The original logic only allowed `.github/`-only commits, but didn't account for personal development environment commits.

## Solution

Updated sync workflows to recognize commits with messages starting with "dev setup" (case-insensitive) as allowed on master, in addition to `.github/`-only commits.

## Changes Made

### 1. Updated Sync Workflows

**Files modified:**
- `.github/workflows/sync-upstream.yml` (automatic hourly sync)
- `.github/workflows/sync-upstream-manual.yml` (manual sync)

**New logic:**
```bash
# Check for "dev setup" commits
DEV_SETUP_COMMITS=$(git log --format=%s upstream/master..origin/master | grep -i "^dev setup" | wc -l)

# Allow merge if:
# - Only .github/ changes, OR
# - Has "dev setup" commits
if [ "$COMMITS_AHEAD" -gt 0 ] && [ "$NON_GITHUB_CHANGES" -gt 0 ]; then
if [ "$DEV_SETUP_COMMITS" -eq 0 ]; then
# FAIL: Code changes outside .github/ that aren't dev setup
exit 1
else
# OK: Dev setup commits are allowed
continue merge
fi
fi
```

### 2. Created Policy Documentation

**New file:** `.github/docs/pristine-master-policy.md`

Documents the "mostly pristine" master policy:
- ✅ `.github/` commits allowed (CI/CD configuration)
- ✅ "dev setup ..." commits allowed (personal development environment)
- ❌ Code changes not allowed (must use feature branches)

## Current Commit Order

```
master:
1. 9a2b895daa0 - Complete Phase 3: Windows builds + fix sync (newest)
2. 1e6379300f8 - Add CI/CD automation: hourly sync, Bedrock AI review
3. e5aa2da496c - dev setup v19
4. 03facc1211b - upstream commits... (oldest)
```

**All three local commits will now be preserved during sync:**
- Commit 1: Modifies `.github/` ✅
- Commit 2: Modifies `.github/` ✅
- Commit 3: Named "dev setup v19" ✅

## Testing

After committing these changes, the next hourly sync should:
1. Detect 3 commits ahead of upstream (including the fix commit)
2. Recognize that they're all allowed (`.github/` or "dev setup")
3. Successfully merge upstream changes
4. Create merge commit preserving all local commits

**Verify manually:**
```bash
# Trigger manual sync
# Actions → "Sync from Upstream (Manual)" → Run workflow

# Check logs for:
# "✓ Found 1 'dev setup' commit(s) - will merge"
# "✓ Successfully merged upstream with local configuration"
```

## Future Updates

When updating your development environment:

```bash
# Make changes
git add .clangd flake.nix .vscode/ .idea/

# IMPORTANT: Start commit message with "dev setup"
git commit -m "dev setup v20: Update IDE and LSP configuration"

git push origin master
```

The sync will recognize this and preserve it during merges.

**Naming patterns recognized:**
- `dev setup v20` ✅
- `Dev setup: Update tools` ✅
- `DEV SETUP - New config` ✅
- `development environment changes` ❌ (doesn't start with "dev setup")

## Benefits

1. **No manual sync resolution needed** for dev environment updates
2. **Simpler workflow** - dev setup stays on master where it's convenient
3. **Clear policy** - documented what's allowed vs what requires feature branches
4. **Automatic detection** - sync workflow handles it all automatically

## What to Commit

```bash
git add .github/workflows/sync-upstream.yml
git add .github/workflows/sync-upstream-manual.yml
git add .github/docs/pristine-master-policy.md
git add .github/DEV_SETUP_FIX.md

git commit -m "Fix sync to allow 'dev setup' commits on master

The sync workflow was failing because the 'dev setup v19' commit
modifies files outside .github/. Updated workflows to recognize
commits with messages starting with 'dev setup' as allowed on master.

Changes:
- Detect 'dev setup' commits by message pattern
- Allow merge if commits are .github/ OR dev setup
- Update merge messages to reflect preserved changes
- Document pristine master policy

This allows personal development environment commits (IDE configs,
debugging tools, shell aliases, etc.) on master without violating
the pristine mirror policy.

See .github/docs/pristine-master-policy.md for details"

git push origin master
```

## Next Sync Expected Behavior

```
Before:
Upstream: A---B---C---D (latest upstream)
Master: A---B---C---X---Y---Z (X=CI/CD, Y=CI/CD, Z=dev setup)

Status: 3 commits ahead, 1 commit behind

After:
Master: A---B---C---X---Y---Z---M
\ /
D-------/

Where M = Merge commit preserving all local changes
```

All three local commits (CI/CD + dev setup) preserved! ✅

---

**Status:** Ready to commit and test
**Documentation:** See `.github/docs/pristine-master-policy.md`
Loading
Loading