docs: bring the READMEs up to date with M2 - #63
Merged
Merged
Conversation
It still said "M0 landed — the scalar core is implemented. The rest is in progress", which was true in July. Since then ddx-core reached 0.2.1 on crates.io, ddxdb 0.1.0 went to PyPI, the DataFusion adapter became publishable, and the JAX oracle landed. The larger gap was that the only way in was a Rust snippet. `pip install ddxdb` now works and is the lowest-friction entry point by a wide margin, so Python leads and Rust follows. Also: - A status table saying where each crate actually is, with links to the registries, replacing per-line milestone tags in the layout block that had to be edited in two places. - A correctness section, because "fail loud, never silently wrong" is the claim this project stands on and nothing on the front page backed it up. Names the three things that do: the JAX oracle, the property soak, and the conventions pinned from both sides. - Dropped `ddx-duckdb/` from the layout, which listed a directory that does not exist. - Badges for the two published artifacts, so the current versions are visible without leaving the page. Every claim verified rather than asserted: the Rust snippet was compiled and run against ddx-core 0.2.1 *from crates.io* (not the workspace), the Python snippet against ddxdb installed from PyPI into a clean venv, the nightly soak against its cron schedule, and both badge URLs return 200. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…xist
The package page justified the client-side rewrite by contrast with
"DuckDB's in-database `ddx('<sql>')` table function", which is an M5
design and ships nowhere. A reader landing on PyPI has no way to know
that, so the sentence reads as a comparison between two available options
and sends them looking for one of them.
The property itself is real and worth keeping — the test proves a temp
table, which exists only on the caller's connection, is visible to the
rewrite. It is now stated on its own terms, with the contrast framed as
the design alternative it is rather than as a feature.
Also documents `supported_functions()`, which the page had not caught up
with — the section was still headed "One other function" — including the
caveat that a name being present does not by itself make an expression
differentiable, so the typed errors remain the general answer.
Docs and one test comment; 130 oracle tests and 51 wheel tests unchanged.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The wheel's metadata carries Apache-2.0 and PyPI shows it in the sidebar, but the rendered long description said nothing — and the sidebar is easy to miss next to a page of prose. Matches how ddx-core's README closes. Verified the claim against the installed wheel rather than the source: License: Apache-2.0, and the OSI classifier agrees. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The contributing guide was Rust-only, and described `ddx-datafusion`, `ddxdb` and `tests/` as scaffolds. All three now ship. Adds a "Working on the Python code" section covering the two Python surfaces and why they are separate environments: the wheel in python/ddxdb (its own cargo workspace, because pyo3 links libpython and folding it in would make every repo-root `cargo build` need a configured interpreter), and the oracle suite in tests/ (a uv project with ddxdb as a path dependency, so `uv sync` builds it and there is no separate develop step to run in the wrong place). The part most worth writing down is `--reinstall-package ddxdb`. A plain `uv sync` audits in 2ms and keeps the stale wheel after a crates/ edit, because uv watches the Python package and the Rust source is not in it — so the suite passes against the engine you had before your change. The guide shows the measured transcript rather than asserting it. Also documents what a Python contributor needs before a failure confuses them: skipped points are normal and the retention rate is asserted; some cases are pinned rather than compared, and "fixing" those by making ddx match JAX is wrong. Beyond Python: - The CI list omitted four of the seven PR gates (the wheel matrix, the JAX oracle, the publish dry-run, the cross-platform wheel build). - The release section covered crates.io only. PyPI is a separate, tag-driven train with its own reasons, including why it is not `on: release` — release-plz cuts a GitHub Release per crate, which would fire it for a ddx-core tag. - Added the standing instruction to read release-plz's generated changelog before merging, since it comes from the PR title and has twice described the wrong package. - `cargo test --workspace` does not run the fuzz suite; the soak is #[ignore]-d. The guide said it did. README: ddx-datafusion published to crates.io, so it gets a badge and the status table now links it rather than calling it "publishable". Every documented command was run from a clean `git archive` checkout rather than this working tree: the wheel flow (51 tests) and the oracle flow (130 tests) both pass from nothing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
alxmrs
commented
Aug 9, 2026
| `maturin develop` step to run in the wrong directory or point at the wrong | ||
| virtualenv. | ||
|
|
||
| > **`--reinstall-package ddxdb` is load-bearing, not defensive.** A plain |
Member
Author
There was a problem hiding this comment.
Can we say it's necessary, not load bearing?
| upgrading is a deliberate `uv lock --upgrade --project tests`, and | ||
| `tests/test_conventions.py` is what reports whether anything JAX promises moved. | ||
|
|
||
| ### How the oracle suite is built, and why it matters when editing it |
Member
Author
There was a problem hiding this comment.
This is too much text about the oracle test suite. Let's omit a lot of this or make it shorter. It's distracting for the purposes of the contrib guide.
Member
Author
There was a problem hiding this comment.
I think we can get rid of this whole section (enumerating the checks). This is pretty self evident, no?
| — see the comments in | ||
| [`.github/workflows/release.yml`](.github/workflows/release.yml), including the | ||
| optional upgrade to crates.io Trusted Publishing. | ||
| - PyPI uses **Trusted Publishing (OIDC)**, so there is no token to leak or |
Member
Author
There was a problem hiding this comment.
We can omit this cause it is only needed to be done once, and it is done.
| ## Status | ||
|
|
||
| **M2 landed and released.** The scalar engine, the DataFusion adapter and the | ||
| Python wheel are all published; ddx runs on DataFusion and DuckDB, and its |
Member
Author
There was a problem hiding this comment.
Wait, we don't have DuckDB support yet.
Member
Author
There was a problem hiding this comment.
Let's at least say "eventually".
- "load-bearing" -> "necessary" for `--reinstall-package ddxdb`. - Cut the oracle-suite explanation down to the two things that would otherwise make a failure confusing, and point at tests/README.md for the rest. The design rationale belongs there, not in a guide someone reads to get set up. - Dropped the enumerated CI checks. The names and their output are on every pull request already, so the list was a second copy that could only go stale. - Dropped the one-time PyPI pending-publisher instructions, which have been carried out. README: the DuckDB claim was too strong. "Runs on DataFusion and DuckDB" reads as two engines with equal support, when only DataFusion has a native adapter — bare grad() in ordinary SQL and through the DataFrame API. DuckDB works through `rewrite_sql` like every other engine: rewrite the text, hand it to your own connection. The status section now draws that line and says the in-database extension comes eventually (M5), which is also what the table's `ddx-duckdb` row has said all along. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The front page still said "M0 landed — the scalar core is implemented. The rest is in progress." That was true in July. Since then
ddx-corereached 0.2.1 on crates.io,ddxdb0.1.0 shipped to PyPI, the DataFusion adapter became publishable, and the JAX oracle landed.The bigger gap: no way in for Python
The only runnable example was a Rust snippet.
pip install ddxdbnow works and is the lowest-friction entry point by a wide margin, so it leads:...followed by the point that makes it interesting —
rewrite_sqlis text in, text out, so it already works with any engine that accepts SQL, not just the ones ddx ships an adapter for. Rust follows.Also
ddx-duckdb/from the layout — it listed a directory that does not exist.Every claim verified, not asserted
A README is the easiest place for aspirational text to survive, so each one was checked:
ddx-core0.2.1 from crates.io, in a scratch crate outside the workspaceddxdbinstalled from PyPI into a clean venv with--no-cachenightly-fuzz.yml, cron0 16 * * *CONTRIBUTING.mdlinkThe Rust check is the one worth calling out: building it outside the workspace is what proves a reader copying it gets working code, rather than code that only compiles here.
Also: the
ddxdbpage on PyPITwo fixes to
python/ddxdb/README.md, which is the long description PyPI renders. They ship with the nextddxdbrelease, not this merge.It compared against a DuckDB function that does not exist. The client-side rewrite was justified by contrast with "DuckDB's in-database
ddx('<sql>')table function" — an M5 design that ships nowhere. A reader landing on PyPI has no way to know that, so it reads as a comparison between two available options and sends them looking for one. The property is real and kept: the test proves a temp table, which exists only on the caller's connection, is visible to the rewrite. It is now stated on its own terms, with the alternative framed as a design rather than a feature.It had not caught up with
supported_functions()— the section was still headed "One other function." Now documented, with the caveat that a name being present does not by itself make an expression differentiable, so the typed errors remain the general answer to "can ddx handle this?".Docs only — no behaviour changes. 130 oracle tests and 51 wheel tests unchanged.
🤖 Generated with Claude Code