diff --git a/README.md b/README.md index 82e4e6e..e72d9d8 100644 --- a/README.md +++ b/README.md @@ -3,44 +3,107 @@ A listing API library for FastAPI, built around small, composable components rather than one large endpoint function. -[![.github/workflows/deploy.yml](https://github.com/danielhasan1/fastapi-listing/actions/workflows/deploy.yml/badge.svg)](https://github.com/danielhasan1/fastapi-listing/actions/workflows/deploy.yml) -[![.github/workflows/tests.yml](https://github.com/danielhasan1/fastapi-listing/actions/workflows/tests.yml/badge.svg)](https://github.com/danielhasan1/fastapi-listing/actions/workflows/tests.yml) ![PyPI - Programming Language](https://img.shields.io/pypi/pyversions/fastapi-listing.svg?color=%2334D058) -[![codecov](https://codecov.io/gh/danielhasan1/fastapi-listing/branch/dev/graph/badge.svg?token=U29ZRNAH8I)](https://codecov.io/gh/danielhasan1/fastapi-listing) [![Downloads](https://static.pepy.tech/badge/fastapi-listing)](https://pepy.tech/project/fastapi-listing) +[![.github/workflows/deploy.yml](https://github.com/danielhasan1/fastapi-listing/actions/workflows/deploy.yml/badge.svg)](https://github.com/danielhasan1/fastapi-listing/actions/workflows/deploy.yml) [![.github/workflows/tests.yml](https://github.com/danielhasan1/fastapi-listing/actions/workflows/tests.yml/badge.svg)](https://github.com/danielhasan1/fastapi-listing/actions/workflows/tests.yml) [![PyPI - Programming Language](https://img.shields.io/pypi/pyversions/fastapi-listing.svg?color=%2334D058)](https://pypi.org/project/fastapi-listing/) [![codecov](https://codecov.io/gh/danielhasan1/fastapi-listing/branch/dev/graph/badge.svg?token=U29ZRNAH8I)](https://codecov.io/gh/danielhasan1/fastapi-listing) [![Downloads](https://static.pepy.tech/badge/fastapi-listing)](https://pepy.tech/project/fastapi-listing) > **Upgrading to 0.4.0?** It's a breaking change **only** if you wrote a custom `Filter`/`Sorter`/`QueryStrategy`/ -> `PaginationStrategy` subclass - the plain `GenericDao` + `generic_filters` + default-strategies flow below is -> unaffected. See [CHANGELOG.md](CHANGELOG.md) for the migration table. +> `PaginationStrategy` subclass — the plain `GenericDao` + `generic_filters` + default-strategies flow below is +> unaffected. See [CHANGELOG.md](https://github.com/danielhasan1/fastapi-listing/blob/dev/CHANGELOG.md) for the +> migration table. -Comes with: -- a predefined set of filters -- a predefined paginator -- a predefined sorter -- SQLAlchemy support out of the box, and a backend-agnostic core so you're not locked into one ORM +## The problem this solves + + +Every listing endpoint starts simple — return some rows, maybe paginate. Then product asks for a filter. +Then three more filters. Then sorting. Then sorting on a joined table. Then a second client needs the +same data with different query-param names. A year in, the endpoint is 200 lines of branching `if` +statements, and nobody wants to touch it. + +`fastapi-listing` keeps that endpoint at roughly the same size on day 1,000 as it was on day 1, by pulling +filtering, sorting, and pagination out into small, independently testable, swappable pieces instead of +letting them accumulate inline. + +## 60-second quickstart + + +``` +pip install fastapi-listing +``` + +```python +# main.py +from fastapi import FastAPI +from sqlalchemy import Column, Integer, String +from sqlalchemy.orm import declarative_base, Session + +from fastapi_listing import FastapiListing, MetaInfo +from fastapi_listing.dao import GenericDao +from fastapi_listing.filters import generic_filters +from fastapi_listing.factory import filter_factory + +Base = declarative_base() +app = FastAPI() + +class Employee(Base): + __tablename__ = "employees" + emp_no = Column(Integer, primary_key=True) + first_name = Column(String(14)) + gender = Column(String(1)) + +class EmployeeDao(GenericDao): + name = "employee" + model = Employee + +# one line per filter you want to expose +filter_mapper = { + "gdr": ("Employee.gender", generic_filters.EqualityFilter), +} +filter_factory.register_filter_mapper(filter_mapper) + +@app.get("/employees") +def get_employees(db: Session): + dao = EmployeeDao(read_db=db) + return FastapiListing(dao=dao).get_response( + MetaInfo(default_srt_on="emp_no", filter_mapper=filter_mapper) + ) +``` + +That's it — `GET /employees` now supports pagination and sorting out of the box, and +`GET /employees?filter=[{"gdr":"M"}]` filters by gender. No hand-rolled query params, no `if filter.gender:` +chains. Add another filter by adding another line to `filter_mapper`; nothing else in the endpoint changes. + +The rest of this README covers what's underneath that: request-format adapters, the full filter/sort/ +pagination vocabulary, class-based listing services, and swapping the database backend entirely. ## Why + - Simplifies designing and maintaining complex listing APIs - Components are independent, reusable, and can be swapped in from anywhere - Well suited to fast-changing requirements -- Not an ORM captive: filters/sorter/paginator are written against a small `QueryContext` contract, not a raw SQLAlchemy `Query` - swap in a different backend without rewriting your filters +- Not an ORM captive: filters/sorter/paginator are written against a small `QueryContext` contract, not a + raw SQLAlchemy `Query` — swap in a different backend without rewriting your filters ## Installing + Using [pip](https://pip.pypa.io/): -```bash +``` pip install fastapi-listing ``` -## Quick example +## Full example -The example below runs against the [MySQL employee sample DB](https://dev.mysql.com/doc/employee/en/). -There are two ways to implement a listing API with this library: **inline** or **class-based**. Both -need a DAO (data access object) class. +The example below runs against the [MySQL employee sample DB](https://dev.mysql.com/doc/employee/en/) and +builds on the quickstart above with a real Pydantic response model and multiple filters. + +There are two ways to implement a listing API with this library: **inline** or **class-based**. Both need +a DAO (data access object) class. ### Inline implementation + ```python # main.py @@ -56,11 +119,9 @@ from fastapi_listing.paginator import ListingPage from fastapi_listing import FastapiListing, MetaInfo from fastapi_listing.dao import GenericDao - Base = declarative_base() app = FastAPI() - class Employee(Base): __tablename__ = 'employees' @@ -77,7 +138,6 @@ class EmployeeDao(GenericDao): name = "employee" model = Employee # SQLAlchemy model class. Not on SQLAlchemy? See "Backend support" below. - class EmployeeListDetails(BaseModel): emp_no: int = Field(alias="empid", title="Employee ID") birth_date: date = Field(alias="bdt", title="Birth Date") @@ -90,7 +150,6 @@ class EmployeeListDetails(BaseModel): orm_mode = True allow_population_by_field_name = True - @app.get("/employees", response_model=ListingPage[EmployeeListDetails]) def get_employees(db: Session): dao = EmployeeDao(read_db=db) @@ -114,10 +173,6 @@ def get_employees(db: Session): ).get_response(MetaInfo(default_srt_on="emp_no")) ``` -That's your first listing response. - -![](https://drive.google.com/uc?export=view&id=1amgrAdGP7WvXfiNlCYJZPC9fz4_1CidE) - If the auto-generated query doesn't fit your use case, override `get_default_read` in the DAO instead: ```python @@ -136,7 +191,6 @@ class EmployeeDao(GenericDao): query = self._read_db.query(Employee) return query - @app.get("/employees", response_model=ListingPage[EmployeeListDetails]) def get_employees(db: Session): dao = EmployeeDao(read_db=db) @@ -144,8 +198,8 @@ def get_employees(db: Session): return FastapiListing(dao=dao).get_response(MetaInfo(default_srt_on="emp_no")) ``` +## Adding client-side features -## Adding client-site features Before adding filters, sorters, or pagination, most existing services need one bit of setup: an adapter that reads your client's actual request-parameter format, unless it already matches FastAPI Listing's @@ -153,7 +207,8 @@ native format exactly. ### Add a custom adapter for reading filter/sorter/paginator parameters -Below is the default implementation - you'll typically extend it with your own parameter format. + +Below is the default implementation — you'll typically extend it with your own parameter format. ```python from typing import Literal @@ -192,15 +247,16 @@ class YourAdapterClass(CoreListingParamsAdapter): # extend to add your own beha @return: List[Optional[dict]] for filter/sort, dict for pagination """ return utils.dictify_query_params(self.dependency.get(key)) - ``` ### Adding filters + Add filters on `Employee` for: -1. **gender** - only employees matching a given gender -2. **date of birth** - employees within a date range -3. **first name** - employees whose first name starts with a given value + +1. **gender** — only employees matching a given gender +2. **date of birth** — employees within a date range +3. **first name** — employees whose first name starts with a given value ```python from fastapi import Request @@ -211,7 +267,6 @@ from fastapi_listing.filters import generic_filters # collection of inbuilt fil from fastapi_listing.factory import filter_factory # register a filter mapper for use from fastapi_listing import MetaInfo, FastapiListing - emp_filter_mapper = { "gdr": ("Employee.gender", generic_filters.EqualityFilter), "bdt": ("Employee.birth_date", generic_filters.MySqlNativeDateFormateRangeFilter), @@ -219,7 +274,6 @@ emp_filter_mapper = { } filter_factory.register_filter_mapper(emp_filter_mapper) - @app.get("/employees", response_model=ListingPage[EmployeeListDetails]) def get_employees(request: Request, db: Session): dao = EmployeeDao(read_db=db) @@ -249,7 +303,8 @@ def get_employees(request: Request, db: Session): ### Breaking it down -**Filter mapper** - the set of filters allowed on this listing API. A request for anything outside this + +**Filter mapper** — the set of filters allowed on this listing API. A request for anything outside this mapper is simply not executed, which keeps clients from probing for fields you didn't intend to expose. `generic_filters` is a collection of inbuilt filters supported by the SQLAlchemy ORM. The mapper's @@ -257,14 +312,13 @@ structure: `{"alias": tuple("sqlalchemy_model.field", filter_implementation)}` -`alias` - what the client sends, so the real column name never has to be exposed. +`alias` — what the client sends, so the real column name never has to be exposed. -`tuple` - the field name and the filter implementation. +`tuple` — the field name and the filter implementation. ```python from fastapi_listing.filters import generic_filters - emp_filter_mapper = { "gdr": ("Employee.gender", generic_filters.EqualityFilter), "bdt": ("Employee.birth_date", generic_filters.MySqlNativeDateFormateRangeFilter), @@ -277,21 +331,19 @@ Register the mapper with the filter factory, at module level: ```python from fastapi_listing.factory import filter_factory - filter_factory.register_filter_mapper(emp_filter_mapper) ``` A client could then request `v1/employees?filter=[{"gdr":"M"}]`, which your adapter parses into -`[{"field":"gdr", "value":{"search":"M"}}]` - if the adapter is given kwargs directly rather than the -request, access them via `self.extra_context`; if it's given the request, access `self.request` -directly. +`[{"field":"gdr", "value":{"search":"M"}}]` — if the adapter is given kwargs directly rather than the +request, access them via `self.extra_context`; if it's given the request, access `self.request` directly. That produces a response filtered to rows where `gender` is `M`. -**Sort mapper** - the set of fields allowed for sorting; a request for anything outside this mapper is +**Sort mapper** — the set of fields allowed for sorting; a request for anything outside this mapper is not permitted. -Structure: `{"alias": "field"}` - omit the model name when sorting on the primary model's own column, or +Structure: `{"alias": "field"}` — omit the model name when sorting on the primary model's own column, or qualify it (`{"alias": "sqlalchemy_model.field"}`) for a joined table's column, same as with filters. ```python @@ -315,14 +367,14 @@ return FastapiListing(request=request, dao=dao).get_response( feature_params_adapter=YourAdapterClass)) ``` -A client could then request `v1/employees?sort={"code":}`, or combine it with a filter - -`v1/employees?filter=[{"gdr":"M"}]&sort={"code":, "type":"asc"}` - and the response would +A client could then request `v1/employees?sort={"code":}`, or combine it with a filter — +`v1/employees?filter=[{"gdr":"M"}]&sort={"code":, "type":"asc"}` — and the response would be sorted by employee code, ascending. **Note**: unlike the filter mapper, the sort mapper doesn't need to be registered with a factory. Pagination works the same way: `v1/employees?pagination={"page":1, "pageSize":10}`, or combined with -filter and sort - `v1/employees?filter=[{"gdr":"M"}]&sort={"code":, "type":"asc"}&pagination={"page":1, "pageSize":10}`. +filter and sort — `v1/employees?filter=[{"gdr":"M"}]&sort={"code":, "type":"asc"}&pagination={"page":1, "pageSize":10}`. That returns a page of 10 items, or however many the client requests via `pageSize`. @@ -339,9 +391,11 @@ return FastapiListing(request=request, dao=dao).get_response( feature_params_adapter=YourAdapterClass)) ``` -### Class-based implementation +## Class-based implementation -The same listing API, structured as a class: + +The same listing API, structured as a class — useful once a listing endpoint has enough mappers and +overrides that keeping them inline in the route function stops being readable: ```python from fastapi import FastAPI @@ -367,7 +421,6 @@ class Title(Base): employee = relationship('Employee') - class EmployeeDao(GenericDao): name = "employee" model = Employee @@ -401,7 +454,6 @@ class EmployeeListingService(ListingService): resp = FastapiListing(self.request, self.dao, pydantic_serializer=EmployeeListDetails).get_response(self.MetaInfo(self)) return resp - @app.get("/employees", response_model=ListingPage[EmployeeListDetails]) def get_employees(db: Session): return EmployeeListingService(read_db=db).get_listing() @@ -412,58 +464,62 @@ for the full list of supported filters. You can also write your own custom filte ## Need something the built-ins don't cover? + You can write a custom: -* Query strategy -* Filter -* Sorter -* Paginator +- Query strategy +- Filter +- Sorter +- Paginator See the customisation section of the docs, after basics and tutorials. -A second, example-focused repo is available [here](https://github.com/danielhasan1/test-fastapi-listing/blob/master/app/router/router.py). +A second, example-focused repo is available +[here](https://github.com/danielhasan1/test-fastapi-listing/blob/master/app/router/router.py). ## Backend support -fastapi-listing ships with SQLAlchemy support by default, but nothing in the Filter/Sorter/Paginator/QueryStrategy -contract is SQLAlchemy-specific. Every one of them is written against a small `QueryContext` interface -(`fastapi_listing/context`), not a raw SQLAlchemy `Query` - `SqlAlchemyQueryContext` is just the default -implementation of it. + +fastapi-listing ships with SQLAlchemy support by default, but nothing in the Filter/Sorter/Paginator/ +QueryStrategy contract is SQLAlchemy-specific. Every one of them is written against a small `QueryContext` +interface (`fastapi_listing/context`), not a raw SQLAlchemy `Query` — `SqlAlchemyQueryContext` is just the +default implementation of it. As proof this isn't SQLAlchemy in disguise, a **ClickHouse** backend ships as a reference implementation -(`fastapi_listing.dao.ClickHouseDao` + `fastapi_listing.context.clickhouse.ClickHouseQueryContext`) - raw +(`fastapi_listing.dao.ClickHouseDao` + `fastapi_listing.context.clickhouse.ClickHouseQueryContext`) — raw parameterized SQL via `clickhouse-driver`, no ORM at all. The same `generic_filters` classes (`EqualityFilter`, `InDataFilter`, ...) and the default `SortingOrderStrategy`/`PaginationStrategy` work against it completely unmodified, because they only ever talk to the `QueryContext`, never to SQLAlchemy directly. -```bash +``` pip install fastapi-listing[clickhouse] ``` -Canonical filters/sort/pagination cover the common case - equality/range/comparison checks on a plain +Canonical filters/sort/pagination cover the common case — equality/range/comparison checks on a plain column, single-column sort, offset/limit pagination. Real queries aren't always that simple, so every escape hatch that exists for SQLAlchemy (`context.native`) has a ClickHouse equivalent, and then some: -* **`ClickHouseQueryContext.from_raw_sql(client=..., sql=..., params=...)`** - the full bypass. Hand-build +- **`ClickHouseQueryContext.from_raw_sql(client=..., sql=..., params=...)`** — the full bypass. Hand-build a query with CTEs, joins, window functions, a table function as the source, whatever the canonical `Op` - vocabulary can't express - using whatever query-building approach you already have - and you still get + vocabulary can't express — using whatever query-building approach you already have — and you still get back a `QueryContext` that canonical filters/sort/pagination can layer on top of, or that you can use completely as-is. -* **`HavingMixin`** - filter on an aggregated field after a `GROUP BY` (`SUM(x) > 100`), same canonical +- **`HavingMixin`** — filter on an aggregated field after a `GROUP BY` (`SUM(x) > 100`), same canonical `Op`s, routed to `HAVING` instead of `WHERE`: `class TotalAbove(HavingMixin, DataGreaterThanFilter): pass`. -* **`order_by_raw(expression)`** - for a compound ordering rule (a tiebreak column, multiple sort keys) +- **`order_by_raw(expression)`** — for a compound ordering rule (a tiebreak column, multiple sort keys) that a single `field, direction` pair can't represent. -* **`add_raw_condition(sql_template, **values)`** - a per-filter escape hatch for a backend-specific SQL - function (a full-text search builtin, an array operator, ...) with no canonical `Op` equivalent - values +- **`add_raw_condition(sql_template, **values)`** — a per-filter escape hatch for a backend-specific SQL + function (a full-text search builtin, an array operator, ...) with no canonical `Op` equivalent — values are still bound through the driver's real parameter binding, never string-formatted into the SQL text. Want a different ORM or database driver (Tortoise, Django ORM, raw psycopg2, pymongo, ...)? Write your own -`QueryContext` + DAO pair the same way `ClickHouseQueryContext`/`ClickHouseDao` do it - see `docs/query.rst`. +`QueryContext` + DAO pair the same way `ClickHouseQueryContext`/`ClickHouseDao` do it — see `docs/query.rst`. Neither SQLAlchemy nor clickhouse-driver are required to install the package; both are opt-in extras. ## Design goals + - A well-defined interface for filter, sorter, and paginator - Dependency injection, for easy testing - Adapters, so an existing client's query-param format doesn't have to change @@ -471,9 +527,11 @@ Neither SQLAlchemy nor clickhouse-driver are required to install the package; bo ## Documentation -Full documentation: https://fastapi-listing.readthedocs.io (a work in progress) +Full documentation: ## Feedback and questions -Feedback and questions are welcome - please [open an issue](https://github.com/danielhasan1/fastapi-listing/issues/new). +[#feedback-and-questions](#feedback-and-questions) + +Feedback and questions are welcome — please [open an issue](https://github.com/danielhasan1/fastapi-listing/issues/new).