Make Filter/Sorter/Paginator/QueryStrategy backend-agnostic; add ClickHouse reference backend (0.4.0) - #21
Merged
Conversation
…kHouse reference backend fastapi-listing's default filter/sort/paginate implementations talked to a raw SQLAlchemy Query directly, and four modules imported sqlalchemy unconditionally at load time, so the library couldn't even be imported without it - let alone plugged into a different ORM or a non-ORM backend. Introduces a QueryContext contract (fastapi_listing.context) that every Filter/Sorter/Paginator/ QueryStrategy now speaks instead, with SqlAlchemyQueryContext as a thin pass-through adapter and ClickHouseQueryContext + ClickHouseDao as a reference non-ORM backend (parameterized raw SQL via clickhouse-driver) proving the abstraction actually generalizes rather than being SQLAlchemy-only in disguise. - generic_filters.CommonFilterImpl -> CanonicalFilter: filters declare a canonical Op (fastapi_listing.ops.Op) and work unchanged against any backend (deprecated alias kept for one release) - PaginationStrategy gains a postprocess(rows, extra_context) hook for post-fetch business logic that previously had no governed home - import fastapi_listing no longer requires SQLAlchemy; clickhouse-driver ships as an opt-in extra, neither is in install_requires - FastapiListingMigrationError replaces bare TypeError/AttributeError when pre-0.4.0 custom Filter/Sorter/QueryStrategy subclasses run unmigrated - loader.py's DAO check now validates against DaoAbstract instead of hardcoding GenericDao, so non-SQLAlchemy DAOs register the same way - typing.Literal (3.8+) guarded with a typing_extensions fallback in the new context modules, matching the existing ctyping.py/service/config.py pattern, so import fastapi_listing still works on Python 3.7 Breaking change for custom Filter/Sorter/QueryStrategy/PaginationStrategy subclasses (signature/return-type change); the default GenericDao + generic_filters + default-strategies flow is unaffected. See CHANGELOG.md. Closes #10
CHANGELOG.md already covered from_raw_sql, HavingMixin, order_by_raw, and add_raw_condition; the README's "Backend support" section and docs/query.rst only described the basic canonical-Op story, undercutting the actual point of these additions - full developer control when canonical filters/sort aren't enough.
409 Conflict is for resource-state conflicts (edit conflicts, duplicate creation) - requesting a filter/sort field outside filter_mapper/sort_mapper isn't that, it's an invalid request parameter. The same file already used 422 for the closely related "filter/sort request itself is malformed" case, so this was an internal inconsistency as well as a spec misuse. Both exception classes are plain fastapi.HTTPException subclasses, so this was always overridable per-app via @app.exception_handler(...) without any library change - the fix here is just to make the library's own default spec-correct, not to add configuration for it.
…al-DB CI - Fixed a real bug in middlewares.py's manager(): the finally block inferred whether *this* call set a session token from the shared ContextVar's current value, which could be a leftover from an earlier, unrelated call (especially with implicit_close=False, which deliberately never resets) - causing an UnboundLocalError. Now tracked locally per call. - Added ~10 test files closing real, previously-untested branches (validated against the real MySQL CI's line-by-line coverage first, not padding): paginator validation/pagination-math, factory registry error paths, the class-based ListingService flow via SQLite (dao_factory fallback, MetaInfo, switch()), loader's page-size check, and an automated check that `import fastapi_listing` actually works without SQLAlchemy installed (previously only verified manually in a throwaway venv). - Added tests/test_clickhouse_real_integration.py: the ClickHouse backend had only ever been validated against a fake client. This runs the same canonical-filter/HAVING/from_raw_sql scenarios against a real ClickHouse server via the real clickhouse-driver package - skips gracefully if none is reachable locally, always provisioned in CI (new docker-compose.dev.yml for local parity, tests.yml now provisions ClickHouse alongside the existing MySQL container). - Removed a dead try/except in context/__init__.py (ClickHouseQueryContext never actually imports clickhouse-driver, so it could never raise) and marked two Protocol-only interface classes `# pragma: no cover` so the coverage report doesn't misrepresent structural type stubs as real gaps. - setup.py: Development Status 5 (Production/Stable) -> 4 (Beta). Shipping a breaking rework with a brand-new, not-yet-battle-tested backend under the same "stable" label as the well-worn SQLAlchemy path overstated it. - Added CONTRIBUTING.md documenting local test setup for both real databases. Full suite verified locally against both a real MySQL instance (the same danielhasan1/mysql_employees_test_db image CI uses) and a real ClickHouse server: 119 passed, 97% coverage.
CLAUDE.md is local AI-assistant working notes - not something to impose on other contributors of a public repo, so it stays untracked rather than committed or even gitignored. .DS_Store is macOS Finder clutter that never should have been tracked in the first place.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
fastapi-listing's Filter/Sorter/Paginator/QueryStrategy contracts were pluggable in theory, but every
shipped default talked to a raw SQLAlchemy
Querydirectly, and four modules imported SQLAlchemyunconditionally at load time — the library couldn't even be imported without it. This introduces
QueryContextas the real seam and shipsClickHouseQueryContext/ClickHouseDaoas a non-ORMreference backend, proving the abstraction actually generalizes rather than just being SQLAlchemy with
extra steps.
queryparams withQueryContextacrossFilter/Sorter/Paginator/QueryStrategy.SqlAlchemyQueryContextis the default;context.nativeis the escape hatch for anything thecanonical
Opvocabulary doesn't cover.CommonFilterImpl→CanonicalFilter(deprecated alias kept, warns for one release). The 14built-in filters now just declare an
Opand work unchanged against any backend.import fastapi_listingno longer requires SQLAlchemy — it's genuinely optional now, matching whatsetup.pyalready claimed.PaginationStrategy.postprocess()— a governed seam for post-fetch logic that previously had nodesignated home.
HavingMixin,order_by_raw(),add_raw_condition(),ClickHouseQueryContext.from_raw_sql()asescape hatches for aggregated filters, compound sorts, and backend-specific SQL builtins — all
validated against a real ClickHouse server, not just a fake client.
count_compile()for grouped ClickHouse queries (was counting per-group, not the total).NotRegisteredApiException: 409 → 422 (correct HTTP semantics for an unregistered filter/sortfield, not a resource-state conflict).
ContextVarleak inmiddlewares.py'smanager()cleanup.FastapiListingMigrationErrorfor anyone upgrading a custom subclass withoutmigrating, instead of a generic traceback.
Breaking only for custom
Filter/Sorter/QueryStrategy/PaginationStrategysubclasses — seeCHANGELOG.mdmigration table. The plain README flow (GenericDao+generic_filters+ defaultstrategies) is unaffected.
Test plan
clients) provisioned in every matrix job.
import fastapi_listingsucceeds.from_raw_sql,HavingMixin,order_by_raw,add_raw_condition)validated against a real local ClickHouse instance.