fix(monitor): resilient registry enumeration — stop silent truncation of the 66k baseline#34
Merged
Merged
Conversation
…y truncates The registry sweep was scanning only a few hundred to a few thousand skills instead of the full ~66k, and had been since 07-07. Root cause: the client retries only connection-level failures, so a single rate-limit/5xx mid-listing threw, and _enumerate_skills silently broke out of pagination and returned the partial list. build_snapshot then treated that partial list as authoritative and dropped every previously-known skill not in it -- one hiccup collapsed the 66k baseline (66,544 on 07-06 -> 91 on 07-07). Two fixes: 1. Resilient pagination: retry each failed listing page (4 attempts, finite backoff) before giving up, and return a completeness flag. Reaching the natural end with every page ok -> complete; a page that still fails after retries -> partial (does not abort the run). 2. Baseline protection: when enumeration is incomplete, carry previously-known skills forward instead of dropping them, so a transient listing failure can no longer shrink the baseline. A complete enumeration still allows genuine removals. Records enumerated_count / enumeration_complete / carried_forward_count on the snapshot for visibility. Adds enumeration-resilience tests (transient-retry, persistent-partial, carry-forward, genuine-removal, empty-listing). Full unit suite (1445) passes. Co-Authored-By: Claude <noreply@anthropic.com>
This was referenced Jul 17, 2026
Ap6pack
added a commit
that referenced
this pull request
Jul 17, 2026
…runcation (#36) * fix(monitor): pace registry enumeration to stop throttle truncation #34 made a truncated enumeration fail safely (honest incomplete flag + baseline carry-forward), but a verification run showed it still only listed ~3,346 of ~66k skills: the ~1,300-page listing fires as fast as possible and trips the registry's sustained-listing throttle partway through, at a variable point (429/940/2,940/3,346 across runs). Pace the listing to ~2 req/s (matching the registry's observed limit) so it stays under the throttle, and extend the per-page retry/backoff to outlast a throttle window. Full enumeration then costs ~11 min, within the run's time box. Tests zero the pacing/backoff via the autouse fixture so the suite stays fast. Monitor tests (49) pass; ruff + mypy clean. True coverage is confirmed by the next run's enumerated_count / enumeration_complete. Co-Authored-By: Claude <noreply@anthropic.com> * feat(monitor): tunable listing page size to cut enumeration requests Enumerating the ~66k registry at the hardcoded page_size=50 is ~1,330 listing requests, the bulk of the throttle exposure. ClawHub has no bulk/export endpoint (public API is /skills paginated, /search, /skills/{slug}[/file], rate-limited to 120 req/min), so fewer/larger pages is the main lever. Add --page-size (default 100, was 50) to 'crawl monitor' and a page_size workflow dispatch input, so a larger page can be tried without code changes (the API returns its own max if we request more). At 100/page enumeration halves to ~660 requests; if the API honors 250, ~270. Complements the pacing/backoff fix in this branch. yaml + CLI help verified; monitor tests (49) pass, ruff + mypy clean. Co-Authored-By: Claude <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The bug
The sweep was scanning only a few hundred to a few thousand skills instead of
the full registry, and had been since 07-07. The intended design — baseline
all skills once (budgeted), then re-enumerate everything each run and re-scan
only new/changed via the diff — was silently broken.
Evidence from the snapshot history on the data branch:
pending=0every run (falsely "complete")Root cause (two compounding bugs):
listing of 66k skills, one rate-limit/5xx/timeout threw, and
_enumerate_skillssilentlybreaked and returned the partial list.build_snapshottreated that partial list as authoritative and droppedevery previously-known skill not in it — so one hiccup wiped the baseline.
The fix
backoff) before giving up, and return a
completeflag. Natural end withevery page ok → complete; a page still failing after retries → partial
(logged loudly, run continues).
skills forward instead of dropping them. A complete enumeration still allows
genuine removals. Records
enumerated_count/enumeration_complete/carried_forward_counton the snapshot for visibility (and the Discordsummary/logs now surface it).
Tests
New enumeration-resilience coverage: transient-page-retry-then-complete,
persistent-page-failure-marks-incomplete, incomplete-carries-forward-baseline,
complete-enumeration-allows-genuine-removal, empty-listing-preserves-baseline.
Full unit suite: 1445 passing; ruff + mypy clean.
After merge
The next full re-scan should re-enumerate the whole ~66k registry and rebuild
the budgeted baseline (draining
pendingover successive runs) instead ofscanning a sliver. Coverage will be visible via
enumerated_count/enumeration_completein each snapshot and the Discord summary.🤖 Generated with Claude Code
Generated by Claude Code