Batch API, plus Fetch/Extract first-class - #13
Open
AnderRV wants to merge 7 commits into
Open
Conversation
Add ZenRowsBatchClient — a synchronous, typed client for the async-job Batch API — and migrate the package to a src/ layout. - Typestate resource handles: JobRef/JobHandle, RunRef/RunHandle, ExportRef/ExportHandle (id-only refs vs loaded handles with .data), with job.run.* (current run) and job.schedule.* facets that disambiguate the two "pause" endpoints. - submit_regular / submit_open / submit_scheduled, typed schedule builders (At/Rate/Calendar), CSV file-input upload, and cursor-free scanners (iter_jobs / iter_runs / iter_results). - Waiters (run / ingest / export) with jittered backoff; transport-level retries of transient failures (429/502/503/504 + network errors) on idempotent requests, honoring Retry-After. - Downloads pulled straight from each result's presigned result_url (no /content endpoint): bulk download_to_dir / download_to_memory, server-side zip download_all_results, and per-task download_task_to_file / download_task_to_memory. - Webhook management (get/put/delete job webhook, test_webhook), HMAC key lifecycle, and offline cost estimation (client.estimate_cost). - RFC 7807 error mapping to BatchAPIError; pydantic v2 models generated from the OpenAPI spec; generated markdown API reference and runnable examples.
* Add fetch()/extract() onto the existing Batch RC branch, current logo Branched from feature/batch-api/rc (tip e0359e1) rather than main - that branch already had a complete, unreleased ZenRowsBatchClient (submit_job/submit_regular/submit_open/submit_scheduled, pagination, CSV upload, webhooks, HMAC rotation, all tested) that's more mature than anything built for Node/Go this session. Building a second batch client on main would have conflicted with this real, already-partially- reviewed work (PRs #10/#11 are spec-sync patches on top of this exact branch). - fetch()/fetch_async() primary, get()/get_async() deprecated aliases. extract()/extract_async(mode=...) added. - 18 new tests targeting return-value behavior (previous tests only asserted on call arguments, never on what the client returns): non-2xx responses pass through unchanged rather than raising, extract() mode is not validated (any string passes through, now documented via test), get()/fetch() produce identical requests, extract() doesn't mutate the caller's params dict, constructor/ close()/context-manager behavior. client.py coverage 87%->97%. - Reworded 2 lowercase "scraper API" mentions in docs/openapi.yaml. - Added the missing logo header + Extract section to this branch's README (predates the main-branch logo fix, different branch/file). Known gap, not addressed here: this branch's batch client has 58 public methods vs. 13 in the new Node/Go reference clients - full parity (scheduling, webhook CRUD, HMAC, CSV, exports, pagination iterators, retry/backoff) is a separate, larger piece of work, not attempted in this session. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DTXXBERPGEnYiYP4Mt3FjF * Fix: brand spelling is Zenrows, not ZenRows in this session's new prose None of this repo's own code had wrong casing (it was already correct pre-session), but the fetch()/extract() docstrings and README sections I added this session used "ZenRows" in prose. Confirmed the correct spelling via zenrows-web commit 9f1f00c6d (PR #1847, merged 2026-08-11): "Brand spelling is Zenrows (capital Z, lowercase r)." Fixed: README title/intro/Extract-section/Batch-intro, and client.py's module/class/method docstrings for fetch()/extract(). Left docs/openapi.yaml's "ZenRows Batch API" title untouched — it mirrors zenrows-web's own docs/api/conveyor/api.md, which itself still says "ZenRows" and hasn't been swept yet; matching the current authoritative source spec matters more here than applying the brand-copy rule to a generated/copied artifact. Class names (ZenRowsClient, ZenRowsBatchClient) are unaffected either way — published exports keep their existing casing per that same PR's own precedent. Re-verified: 117/117 tests pass. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DTXXBERPGEnYiYP4Mt3FjF * Fall back to Autoparse when Extract's auto mode hits AUTH010 extract(mode="auto") (the default) is a domain-gated open beta - when the target domain isn't enabled yet, the API returns 402 with code AUTH010. Catch that and retry once with autoparse=True instead of raising, matching the CLI's own extract adapter. Opt out with fallback_to_autoparse=False on both extract() and extract_async(). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DTXXBERPGEnYiYP4Mt3FjF * Fix wording: Extract/Batch are beta, not invite-gated "Private beta" implied an invite/enablement step for the whole product. Neither Extract nor Batch actually gate that way — confirmed live, a plain API key can call both immediately. Extract does gate per-domain (AUTH010), which is a separate, narrower fact already covered by the fallback logic above; it isn't "you need an invite to use Extract at all." Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DTXXBERPGEnYiYP4Mt3FjF --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
# Conflicts: # requirements.txt
1 task
Targets that need js_render/premium_proxy used to fail extract() with a 400 REQS002 unless the caller passed those manually - Zoopla listing pages are a real example. adaptive_stealth=True (default) sends mode="auto" at the wire-param level (unrelated to this method's own `mode` argument, which picks the extract contract) on both the extract attempt and the Autoparse fallback. Pass adaptive_stealth=False to disable and set js_render/premium_proxy yourself. Verified live: extract(zoopla_url) now succeeds with zero extra params, and mode=auto doesn't interfere with AUTH010 detection on domains that are still gated. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DTXXBERPGEnYiYP4Mt3FjF
No GitHub Actions existed in this repo at all before this. Runs make check (ruff lint + format --check, ty typecheck) and make test (pytest) on every PR and on push to main/feature/batch-api/rc. Deliberately scoped to lint/checks/tests only - no publish automation. See the separate investigation into what a PyPI release workflow would need before that gets built. Claude-Session: https://claude.ai/code/session_01DTXXBERPGEnYiYP4Mt3FjF Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
ZenRowsBatchClient) — async job/run/task lifecycle via Conveyor: submit (open/regular/scheduled), pagination, CSV upload, webhooks, HMAC key rotation, results export, cost estimation. Newsrc/layout (pyproject.toml/hatchling/uv), replacing the old flatzenrows/package andrequirements.txt.ZenRowsClient—fetch()/fetch_async()(primary;get()/get_async()kept as deprecated aliases),extract()/extract_async(mode=...)for Extract (AI-powered structured extraction, beta).extract(mode="auto")falls back to Autoparse once on a 402AUTH010(domain not yet enabled), matching the CLI's own extract adapter — opt out withfallback_to_autoparse=False.Test plan
pytest— 123/123 passruff check— cleansubmit_open()— all verified end to endrequirements.txtmodify/delete conflict againstmain(dependabot's wheel bump landed onmainafter this branch deleted the file in favor ofpyproject.toml/uv— took the deletion, nothing else conflicted)