feat(wallet): accelerate storage synchronization

[BraydenLangley]

Keep this pull request in draft until local validation is complete. After
every push, wait for all applicable checks on the exact head to finish and
fix every failure before requesting review or calling the work complete.

Program and scope

• Tracker or issue: p2ppsr/metanet-client-desktop#42
• Program gate(s) advanced: practical remote-to-local wallet replication, exact progress/ETA, backward-compatible provider rollout
• Why this change is needed: an isolated desktop benchmark found remote getSyncChunk reads dominated replication time. A legacy provider also reproduced HTTP 413 when the requested rough page size fit the request contract but the serialized JSON-RPC response exceeded the service ceiling.
• Explicitly out of scope: changing WalletStorageManager's global sync lock or allowing normal writes to interleave with offset-based replication. That requires a cursor/keyset or consistent-snapshot protocol so mutations cannot shift offsets. No production service rollout or npm publication is performed here.
• Exact head SHA reviewed: 74ee8521c9

Impact

• No public package source or manifest changed
• Public package source or manifest changed; affected packages are listed below
• Infrastructure source, dependency, image, or deployment configuration changed
• Public API, exports, types, runtime targets, or browser/mobile behavior changed
• Security-sensitive boundary changed
• Documentation or examples changed

Affected packages/services and intended versions (publication occurs only through the release workflow after approval):

• @bsv/wallet-toolbox 2.11.0
• @bsv/wallet-toolbox-client 2.11.0
• @bsv/wallet-toolbox-mobile 2.11.0
• Storage services must consume the released toolbox and run its normal Knex migration before clients receive the faster source reads and new SQL indexes.

Implementation

• Fill sync pages with bounded adaptive source reads: 10 records initially, then size-aware 8x growth capped at 250. A 250-record small-transaction fixture uses 3 source queries (10, 80, 160) instead of 25.
• Add (userId, provenTxId) and (userId, txid) transaction indexes. SQLite EXPLAIN tests prove both plans use the new indexes.
• Commit each received page and its checkpoint in one provider transaction. Failed pages roll back without advancing the durable checkpoint, and IndexedDB abort cleanup preserves the original error.
• Add opt-in source totals for exact progress and ETA. Counts are best-effort and execute only when requested; legacy peers remain wire-compatible.
• Carry the writer-local syncStateId into page requests and use it to disambiguate legacy duplicate checkpoints.
• Make provider registration unique by user and provider identity rather than display name.
• For remote getSyncChunk only, recognize HTTP 413, halve the requested rough response budget, retry at most four times, and remember the successful limit for later pages. The original request object is not mutated, unrelated failures are not retried, and write RPCs are never replayed.

Verification

• Focused remoting tests: 13/13 passed, including successful 413 recovery, remembered page limits, no retry for unrelated 503 failures, and bounded exhaustion.
• Repository health: 184/184 controls passed; documentation, governance, and generated API migration records are current.
• Toolbox lint, typecheck, and build passed.
• Browser contract passed locally: Vite 1,608,681 raw / 379,257 gzip / 297,202 Brotli; esbuild 1,253,841 raw / 344,592 gzip / 277,563 Brotli. Hosted Linux measured esbuild gzip at 345,525 bytes, covered by the reviewed 345,600-byte ceiling.
• Mobile contract passed locally: Metro 1,661,661 raw / 419,463 gzip / 326,436 Brotli; Hermes 3,369,326 raw. Hosted Linux measured Hermes gzip at 1,366,854 bytes, covered by the reviewed 1,367,000-byte ceiling.
• A linked desktop consumer reproduced the real HTTP 413 against an existing provider, automatically retried with a smaller page, and then sustained successful local proof and transaction page commits until the fetch-only test was intentionally stopped. The partial checkpoint remained resumable and SQLite quick_check passed.
• Consumer UX observation: the desktop progress dialog did not repaint after its first visible chunk even while durable counts advanced, and its Cancel button was disabled during the operation. Background operation state, repainting progress, and real between-page cancellation remain required consumer-side follow-ups; this PR does not claim to solve them.
• Hosted CI: exact head 74ee8521c9 passed 32 checks with 7 expected skips, 0 failures, and 0 pending checks. The PR remains draft for operator review.
• Complete local diff self-review performed for correctness, compatibility, retry safety, artifacts, docs, and accidental operational-data disclosure.
• All applicable checks are terminal and successful on exact head 74ee8521c9

Security and dependencies

• No dependency or lockfile change
• Changelog, runtime relevance, peer compatibility, and transitive impact reviewed
• No write RPC is automatically retried
• No new override, advisory dismissal, suppression, or skipped test
• Public diff contains no test-provider hostname, wallet identity, database path, SSH topology, or credentials

Compatibility

• Older clients omit includeTotals and syncStateId; older providers ignore the additive request fields.
• The 413 fallback is client-only and applies to an idempotent sync read. Providers do not need to implement a new endpoint.
• A provider that fits the original page budget sees exactly one request and no behavior change.
• A provider returning another status or a non-network 413-shaped application error is not retried.

Release and operations

• No npm publication was performed from a workstation or from this PR
• Documentation, changelog, migration, and release notes are current
• Rollback remains wire-compatible; reverting clients removes retry/totals support, while reverting services forfeits the source-query and index improvements

Rollout remains staged: review and merge the packages, publish through the governed release workflow, rebuild storage services against the release, validate a non-production remote-to-local sync, then enable the consumer UI. Do not advertise production local backup until the consumer progress/background/cancel follow-ups are validated.

Completion evidence

• The linked tracker is updated only for work proved by merged code and passing checks
• Review conversations are resolved
• Documentation, migration notes, release notes, and operator guidance are current or concretely not applicable
• Exact-head hosted checks are terminal and green

[codecov]

Codecov Report

❌ Patch coverage is 94.73684% with 4 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...-toolbox/src/storage/remoting/StorageClientBase.ts	87.50%	0 Missing and 2 partials ⚠️
...s/wallet/wallet-toolbox/src/storage/StorageKnex.ts	85.71%	0 Missing and 1 partial ⚠️
...wallet/wallet-toolbox/src/storage/StorageReader.ts	0.00%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud