fix(monitor): pace registry enumeration to stop throttle truncation#36
Merged
Conversation
#34 made a truncated enumeration fail safely (honest incomplete flag + baseline carry-forward), but a verification run showed it still only listed ~3,346 of ~66k skills: the ~1,300-page listing fires as fast as possible and trips the registry's sustained-listing throttle partway through, at a variable point (429/940/2,940/3,346 across runs). Pace the listing to ~2 req/s (matching the registry's observed limit) so it stays under the throttle, and extend the per-page retry/backoff to outlast a throttle window. Full enumeration then costs ~11 min, within the run's time box. Tests zero the pacing/backoff via the autouse fixture so the suite stays fast. Monitor tests (49) pass; ruff + mypy clean. True coverage is confirmed by the next run's enumerated_count / enumeration_complete. Co-Authored-By: Claude <noreply@anthropic.com>
Enumerating the ~66k registry at the hardcoded page_size=50 is ~1,330 listing
requests, the bulk of the throttle exposure. ClawHub has no bulk/export
endpoint (public API is /skills paginated, /search, /skills/{slug}[/file],
rate-limited to 120 req/min), so fewer/larger pages is the main lever.
Add --page-size (default 100, was 50) to 'crawl monitor' and a page_size
workflow dispatch input, so a larger page can be tried without code changes
(the API returns its own max if we request more). At 100/page enumeration
halves to ~660 requests; if the API honors 250, ~270.
Complements the pacing/backoff fix in this branch. yaml + CLI help verified;
monitor tests (49) pass, ruff + mypy clean.
Co-Authored-By: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #34
#34 made a truncated enumeration fail safely — it now records an honest
enumeration_complete: falseand carries the baseline forward instead ofdropping it. But a verification run (rules-only, $0) proved it still only listed
3,346 of ~66k skills.
Root cause: the full listing is ~1,300 pages, fired as fast as possible.
The registry throttles sustained listing, so it trips partway — at a variable
point (429 / 940 / 2,940 / 3,346 across runs), the signature of throttling, not
a fixed bug. #34's short retries pushed further but still gave up before the
throttle window cleared.
Fix
_PAGE_DELAY = 0.5), matching theregistry's observed limit, so sustained listing stays under the throttle.
Full enumeration then costs ~11 min, within the run's time box.
throttle window if one is still hit.
Tests zero the pacing/backoff via the autouse fixture so the suite stays fast.
Monitor tests (49) pass; ruff + mypy clean.
Verification
This is an environment I can't reproduce locally (the egress proxy blocks the
registry), so the real proof is the next run's
enumerated_count/enumeration_complete. After merge I'll trigger another free rules-only runand confirm it reaches ~66k with
enumeration_complete: true.🤖 Generated with Claude Code
Generated by Claude Code