Batch API, plus Fetch/Extract first-class

.github/workflows/ci.yml

@@ -0,0 +1,34 @@
+name: CI
+
+on:
+  pull_request:
+    types: [opened, synchronize, reopened]
+  push:
+    branches:
+      - main
+      - feature/batch-api/rc
+
+permissions:
+  contents: read
+
+jobs:
+  check:
+    name: "Lint, typecheck, and test"
+    runs-on: ubuntu-latest
+    steps:
+      - name: "Fetch source code"
+        uses: actions/checkout@v4
+
+      - name: "Install uv"
+        uses: astral-sh/setup-uv@v3
+        with:
+          enable-cache: true
+
+      - name: "Install dependencies"
+        run: uv sync --all-extras
+
+      - name: "Lint, format check, and typecheck"
+        run: make check
+
+      - name: "Unit tests"
+        run: make test

.gitignore

@@ -19,3 +19,16 @@ pip-delete-this-directory.txt
 
 .idea/*
 .vscode/
+
+# uv / venv
+.venv/
+.uv-cache/
+
+# Test + ruff caches
+.pytest_cache/
+.ruff_cache/
+.coverage
+htmlcov/
+
+# Local scratch
+.DS_Store

DEVELOPMENT.md

@@ -1,29 +1,80 @@
-## Development
+# Development
 
-Useful commands for development and publishing.
+This project uses [uv](https://docs.astral.sh/uv/) for env + dependencies,
+[ruff](https://docs.astral.sh/ruff/) for lint + format, and
+[ty](https://docs.astral.sh/ty/) for static type checking. Source lives
+under `src/zenrows/`; tests under `tests/`.
 
-### Install
+## First-time setup
 
-`make install` will install the dependencies from the `requirements.txt` file.
+```bash
+uv sync --all-extras
+```
 
-### Build
+Creates `.venv/` and installs everything in `pyproject.toml`, including
+dev tools. After that, prefix commands with `uv run …` or use the
+Makefile targets below.
 
-`make build` generates the distribution packages. It will not delete previous builds. Remember to change the `__version__` before publishing or it will fail.
+## Layout
 
-### Clean
+```
+src/zenrows/
+├── __init__.py               # re-exports both clients
+├── client.py                 # ZenRowsClient (legacy sync scraper)
+└── batch/
+    ├── __init__.py           # ZenRowsBatchClient + key models
+    ├── client.py             # hand-written typed facade
+    ├── _transport.py         # httpx wrapper, RFC 7807 → exceptions
+    ├── errors.py             # BatchAPIError, ProblemDetail
+    └── models.py             # GENERATED — pydantic v2 (do not edit)
+```
 
-`make clean` removes previous builds and cache files.
+The Batch SDK is split deliberately:
 
-### Lint
+| File           | Owner             | Regenerate?       |
+|----------------|-------------------|-------------------|
+| `models.py`    | datamodel-codegen | `make generate`   |
+| `client.py`    | hand-written      | never auto        |
+| `_transport.py`| hand-written      | never auto        |
+| `errors.py`    | hand-written      | never auto        |
 
-`make lint` runs the linter (`flake8`) on the source and test files.
+This way the wire types stay in lockstep with the OpenAPI document
+while the ergonomic surface (method names, helpers, retries, URL
+override) stays in our control.
 
-### Test
+## Common tasks
 
-`make test` runs all the tests.
+| Make target     | What it does |
+|-----------------|--------------|
+| `make sync`     | `uv sync --all-extras` |
+| `make test`     | `uv run pytest` |
+| `make check`    | `ty check` + `ruff check` + `ruff format --check` (CI mode) |
+| `make typecheck`| `ty check src` (static types; `models.py` excluded) |
+| `make lint`     | `ruff check --fix` |
+| `make format`   | `ruff format` |
+| `make generate` | Re-emit `src/zenrows/batch/models.py` from `docs/openapi.yaml` |
+| `make build`    | Build wheel + sdist via hatchling |
+| `make clean`    | Drop caches + build outputs |
 
-### Upload to PyPI
+## Refreshing the OpenAPI spec
 
-`python -m twine upload dist/*` uploads the latest build to PyPI. It will upload the whole `dist` folder, failing if there was a previous version. Run the `clean` command on those cases. Upload attempts of existing versions will fail with a `File already exists` error.
+`docs/openapi.yaml` is the SDK-local copy of the spec. `make generate`
+reads it to emit the models. To refresh after a backend spec change:
 
-For uploading to the test repository, use `python -m twine upload --repository testpypi dist/*`. The same restrictions apply.
+1. Copy the updated spec into `docs/openapi.yaml`.
+2. Run `make generate`.
+3. Run `make check && make test`.
+4. If the wire shape changed, update `src/zenrows/batch/client.py`
+   so the facade method signatures still typecheck.
+
+## Publishing
+
+```bash
+make clean
+make build                          # produces dist/*.whl + dist/*.tar.gz
+uv run twine upload dist/*          # or test PyPI:
+uv run twine upload --repository testpypi dist/*
+```
+
+Bump `version` in `pyproject.toml` and `src/zenrows/__version__.py`
+together before each release.

Makefile

@@ -1,18 +1,69 @@
-.PHONY: install build clean lint test
+.PHONY: install sync test lint format typecheck check generate docs clean build
 
-install:
-	pip install -r requirements.txt
+# Bootstrap: install + dev deps, build the local venv.
+install sync:
+	uv sync --all-extras
 
-build:
-	python setup.py sdist bdist_wheel
+# Run the suite.
+test:
+	uv run pytest
 
-clean:
-	python setup.py clean
-	rm -rf dist build zenrows.egg-info .pytest_cache
-	find . -name '__pycache__' -delete -o -name '*.pyc' -delete
+# Static type check (ty — Astral). Shipped surface only; tests are
+# covered by the suite. Generated models.py is excluded in pyproject.
+typecheck:
+	uv run ty check src
 
+# Lint + format + type check (CI mode — no fixes).
+check: typecheck
+	uv run ruff check src/ tests/
+	uv run ruff format --check src/ tests/
+
+# Lint + format (writes fixes).
 lint:
-	flake8 --config flake8 setup.py tests zenrows
+	uv run ruff check --fix src/ tests/
+format:
+	uv run ruff format src/ tests/
 
-test:
-	python -m pytest tests
+# Regenerate the pydantic v2 models from the backend's canonical spec.
+# docs/openapi.yaml is the SDK-local copy of the spec; refresh it from the
+# backend when the API changes.
+# The HTTP client + facade are HAND-WRITTEN in src/zenrows/batch/client.py;
+# only the type definitions come from this command.
+generate:
+	uv run datamodel-codegen \
+		--input docs/openapi.yaml \
+		--input-file-type openapi \
+		--output src/zenrows/batch/models.py \
+		--output-model-type pydantic_v2.BaseModel \
+		--target-python-version 3.10 \
+		--use-schema-description \
+		--use-field-description \
+		--use-double-quotes \
+		--field-constraints \
+		--use-standard-collections \
+		--use-union-operator \
+		--enum-field-as-literal one \
+		--collapse-root-models \
+		--use-annotated \
+		--capitalise-enum-members \
+		--reuse-model \
+		--use-default
+
+# Regenerate the markdown API reference (docs/batch-client-reference.md) from
+# the SDK's docstrings via pydoc-markdown (ephemeral — no permanent dep). The
+# builder relabels internal module headers to public section titles and strips
+# the `zenrows.batch._x.` qualifiers, so the private `_module` layout never
+# leaks into the customer-facing reference. See scripts/build_reference.py.
+docs:
+	@mkdir -p docs
+	@uv run --with pydoc-markdown python scripts/build_reference.py > docs/batch-client-reference.md
+	@echo "wrote docs/batch-client-reference.md ($$(wc -l < docs/batch-client-reference.md) lines)"
+
+# Clean build artifacts + caches.
+clean:
+	rm -rf dist build *.egg-info src/*.egg-info .pytest_cache .ruff_cache
+	find . -type d -name __pycache__ -prune -exec rm -rf {} +
+
+# Build wheel + sdist via hatchling.
+build:
+	uv build