query: megakey Electrum/Esplora join; SH extents (schema 19) - #162
Merged
Conversation
Electrum/Esplora history loaded each create with serial get_tx_meta_and_outputs. Megakeys paid one idx+body pread per create. Drive the same join through load_creates_once in 4096-fk waves and fill txids from txid.body (page-grouped), leaving the pin path's zero-txid contract unchanged. Co-authored-by: Cursor <cursoragent@cursor.com>
History looked up spenders via txid (tx.head) and loaded the spender Class A body just for identity. Wave spent.body 8-byte peeks at known abs offsets, then fill spender txids from txid.body. Multi-list stays the rare create-fk walk. Co-authored-by: Cursor <cursoragent@cursor.com>
History, balance, listunspent, and chain_stats each walked creates (and spends) separately. Drive all four from join_creates_and_spends so a megakey pays one waved Class A + spent.body session per call. Co-authored-by: Cursor <cursoragent@cursor.com>
Query walked megakey page chains with one 4 KiB pread per page. When last = first + (n-1)*4096, one completion-session (or pread) span fills the chain; linked walk remains the fragmented fallback. entries() uses a single locate_head so last-page first recovery is not paid twice. Co-authored-by: Cursor <cursoragent@cursor.com>
Name the waved idx→body + spent 8 B batch on the process RBITCOIN_IO session. --api-log wall_ms stays the full handler; --log-level debug emits sh_join: pages/class_a/spends when the join exceeds 10 ms. Co-authored-by: Cursor <cursoragent@cursor.com>
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
rbitcoin-consensus structure_with_pres_skips_from_tx races global plan_stamp_sub_stats atomics with parallel structure_meters_split. Co-authored-by: Cursor <cursoragent@cursor.com>
Megakey extents need a new pack8 mode. 18 datadirs rewrite meta even when tx.head / scripthash* are populated; 17 occupied indexes still refuse. 13–18 table headers stay openable. Co-authored-by: Cursor <cursoragent@cursor.com>
Mode 11 stores last_page; that page's 24 B header holds extent_base and extent_n. Bad n=0 or unaligned base is Corrupt. Mode 10 pack8 is unchanged. Co-authored-by: Cursor <cursoragent@cursor.com>
Bulk and tip-first megakey packs allocate used pages at bump with no slack. Last page is ver=2; pack8 is mode 11. entries() span-reads extent_n. The next key packs at extent_end. Co-authored-by: Cursor <cursoragent@cursor.com>
Overflow alloc_page and sets next; extent_n stays unless the new page is still glued to bump. entries() span-reads the prefix then walks the tail. A second overflow adds another linked page. Co-authored-by: Cursor <cursoragent@cursor.com>
Shard and ovf bodies are slow-growing. Align64k rounds need up to 64 KiB instead of Class A 64–256 MiB slabs. Idx grow_tight (1 MiB) is unchanged. Co-authored-by: Cursor <cursoragent@cursor.com>
pack8 mode 11 last page stores extent_base/n. Query span-reads that run then a linked tail. Drop last=first+(n-1)*4096 as the contiguity check. SH body grow is 64 KiB. Co-authored-by: Cursor <cursoragent@cursor.com>
rearden-grok
Bot
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Electrum
get_history/get_balance/listunspentand Esplora address txs/utxo/chain_statsexpand Class B create lists through Class A + spends. That used to be ~3 serial preads per create.One owner in
rbitcoin-query: wavedload_creates_once(IdxBodyMode::Outs) +txids_get_many+tx_spent_range_batch/get_spender_meta_at_abs_batchon the processRBITCOIN_IOsession. History, balance, listunspent, and chain_stats share that join.Megakey extents (schema 19, this follow-on): new megakeys write pack8 mode 11. The last page is
ver=2and storesextent_base+extent_n(tight used count, no slack).entries()span-reads that run. Tip overflowalloc_pages a 4 KiB page and links it after the extent (extent_nunchanged) unless the bump is still glued to the extent end. Schema-18 mode 10 leftovers stay a linked walk. Occupied 18 datadirs rewritemetato 19. SH shard / ovf bodies grow in 64 KiB steps, not Class A 64–256 MiB slabs.Protocol results stay identical. Confirm idx-body / spend-annotate machines are called, not flattened. No
last = first + (n−1)×4096guess as the contiguity owner.Test plan
scripthash_historyincrementsbody_ok_readsby create countchain_stats.tx_countmatches unique history txidsspenders(txid)/get_txgone from scripthash query methodsmetabecomes 19; mode 10entries()still matchver=2last-page codec (badextent_n==0/ unaligned base is Corrupt)extent_nequals page count, next key packs atextent_end,entriesmatch,sh_page_chain_ios() ≤ 2(RBITCOIN_IOsession andpread)last_pageis a tail,extent_nunchanged, second overflow adds another linked page; glued HWM bumpsextent_nin placegrow_tight1 MiB still holdsfmt,deny,clippy,test,multinode,coverage)