Skip to content

docs: bring the READMEs up to date with M2 - #63

Merged
alxmrs merged 5 commits into
mainfrom
docs-readme-m2
Aug 9, 2026
Merged

docs: bring the READMEs up to date with M2#63
alxmrs merged 5 commits into
mainfrom
docs-readme-m2

Conversation

@alxmrs

@alxmrs alxmrs commented Aug 9, 2026

Copy link
Copy Markdown
Member

The front page still said "M0 landed — the scalar core is implemented. The rest is in progress." That was true in July. Since then ddx-core reached 0.2.1 on crates.io, ddxdb 0.1.0 shipped to PyPI, the DataFusion adapter became publishable, and the JAX oracle landed.

The bigger gap: no way in for Python

The only runnable example was a Rust snippet. pip install ddxdb now works and is the lowest-friction entry point by a wide margin, so it leads:

import ddxdb
ddxdb.rewrite_sql("SELECT grad(sin(x), x) AS d FROM t")
# 'SELECT (cos(x)) AS d FROM t'

...followed by the point that makes it interesting — rewrite_sql is text in, text out, so it already works with any engine that accepts SQL, not just the ones ddx ships an adapter for. Rust follows.

Also

  • A status table saying where each crate actually is, with registry links. This replaces per-line milestone tags in the layout block, which meant status lived in two places and drifted in both.
  • A correctness section. "Fail loud, never silently wrong" is the claim this project stands on, and nothing on the front page backed it up. It now names the three things that do: the JAX oracle, the property soak, and the conventions pinned from both sides where ddx and JAX differ on purpose.
  • Dropped ddx-duckdb/ from the layout — it listed a directory that does not exist.
  • Badges for the two published artifacts, so current versions are visible without leaving the page.

Every claim verified, not asserted

A README is the easiest place for aspirational text to survive, so each one was checked:

claim how
the Rust snippet compiled and run against ddx-core 0.2.1 from crates.io, in a scratch crate outside the workspace
the Python snippet run against ddxdb installed from PyPI into a clean venv with --no-cache
"a soak that runs nightly" nightly-fuzz.yml, cron 0 16 * * *
badge URLs both return 200
CONTRIBUTING.md link exists

The Rust check is the one worth calling out: building it outside the workspace is what proves a reader copying it gets working code, rather than code that only compiles here.

Also: the ddxdb page on PyPI

Two fixes to python/ddxdb/README.md, which is the long description PyPI renders. They ship with the next ddxdb release, not this merge.

It compared against a DuckDB function that does not exist. The client-side rewrite was justified by contrast with "DuckDB's in-database ddx('<sql>') table function" — an M5 design that ships nowhere. A reader landing on PyPI has no way to know that, so it reads as a comparison between two available options and sends them looking for one. The property is real and kept: the test proves a temp table, which exists only on the caller's connection, is visible to the rewrite. It is now stated on its own terms, with the alternative framed as a design rather than a feature.

It had not caught up with supported_functions() — the section was still headed "One other function." Now documented, with the caveat that a name being present does not by itself make an expression differentiable, so the typed errors remain the general answer to "can ddx handle this?".

Docs only — no behaviour changes. 130 oracle tests and 51 wheel tests unchanged.

🤖 Generated with Claude Code

alxmrs and others added 2 commits August 9, 2026 13:43
It still said "M0 landed — the scalar core is implemented. The rest is in
progress", which was true in July. Since then ddx-core reached 0.2.1 on
crates.io, ddxdb 0.1.0 went to PyPI, the DataFusion adapter became
publishable, and the JAX oracle landed.

The larger gap was that the only way in was a Rust snippet. `pip install
ddxdb` now works and is the lowest-friction entry point by a wide margin,
so Python leads and Rust follows.

Also:

- A status table saying where each crate actually is, with links to the
  registries, replacing per-line milestone tags in the layout block that
  had to be edited in two places.
- A correctness section, because "fail loud, never silently wrong" is the
  claim this project stands on and nothing on the front page backed it
  up. Names the three things that do: the JAX oracle, the property soak,
  and the conventions pinned from both sides.
- Dropped `ddx-duckdb/` from the layout, which listed a directory that
  does not exist.
- Badges for the two published artifacts, so the current versions are
  visible without leaving the page.

Every claim verified rather than asserted: the Rust snippet was compiled
and run against ddx-core 0.2.1 *from crates.io* (not the workspace), the
Python snippet against ddxdb installed from PyPI into a clean venv, the
nightly soak against its cron schedule, and both badge URLs return 200.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…xist

The package page justified the client-side rewrite by contrast with
"DuckDB's in-database `ddx('<sql>')` table function", which is an M5
design and ships nowhere. A reader landing on PyPI has no way to know
that, so the sentence reads as a comparison between two available options
and sends them looking for one of them.

The property itself is real and worth keeping — the test proves a temp
table, which exists only on the caller's connection, is visible to the
rewrite. It is now stated on its own terms, with the contrast framed as
the design alternative it is rather than as a feature.

Also documents `supported_functions()`, which the page had not caught up
with — the section was still headed "One other function" — including the
caveat that a name being present does not by itself make an expression
differentiable, so the typed errors remain the general answer.

Docs and one test comment; 130 oracle tests and 51 wheel tests unchanged.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@alxmrs alxmrs changed the title docs: bring the README up to date with M2 docs: bring the READMEs up to date with M2 Aug 9, 2026
alxmrs and others added 2 commits August 9, 2026 13:52
The wheel's metadata carries Apache-2.0 and PyPI shows it in the sidebar,
but the rendered long description said nothing — and the sidebar is easy
to miss next to a page of prose. Matches how ddx-core's README closes.

Verified the claim against the installed wheel rather than the source:
License: Apache-2.0, and the OSI classifier agrees.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The contributing guide was Rust-only, and described `ddx-datafusion`,
`ddxdb` and `tests/` as scaffolds. All three now ship.

Adds a "Working on the Python code" section covering the two Python
surfaces and why they are separate environments: the wheel in
python/ddxdb (its own cargo workspace, because pyo3 links libpython and
folding it in would make every repo-root `cargo build` need a configured
interpreter), and the oracle suite in tests/ (a uv project with ddxdb as
a path dependency, so `uv sync` builds it and there is no separate
develop step to run in the wrong place).

The part most worth writing down is `--reinstall-package ddxdb`. A plain
`uv sync` audits in 2ms and keeps the stale wheel after a crates/ edit,
because uv watches the Python package and the Rust source is not in it —
so the suite passes against the engine you had before your change. The
guide shows the measured transcript rather than asserting it.

Also documents what a Python contributor needs before a failure confuses
them: skipped points are normal and the retention rate is asserted; some
cases are pinned rather than compared, and "fixing" those by making ddx
match JAX is wrong.

Beyond Python:

- The CI list omitted four of the seven PR gates (the wheel matrix, the
  JAX oracle, the publish dry-run, the cross-platform wheel build).
- The release section covered crates.io only. PyPI is a separate,
  tag-driven train with its own reasons, including why it is not
  `on: release` — release-plz cuts a GitHub Release per crate, which
  would fire it for a ddx-core tag.
- Added the standing instruction to read release-plz's generated
  changelog before merging, since it comes from the PR title and has
  twice described the wrong package.
- `cargo test --workspace` does not run the fuzz suite; the soak is
  #[ignore]-d. The guide said it did.

README: ddx-datafusion published to crates.io, so it gets a badge and the
status table now links it rather than calling it "publishable".

Every documented command was run from a clean `git archive` checkout
rather than this working tree: the wheel flow (51 tests) and the oracle
flow (130 tests) both pass from nothing.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Comment thread CONTRIBUTING.md Outdated
`maturin develop` step to run in the wrong directory or point at the wrong
virtualenv.

> **`--reinstall-package ddxdb` is load-bearing, not defensive.** A plain

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we say it's necessary, not load bearing?

Comment thread CONTRIBUTING.md Outdated
upgrading is a deliberate `uv lock --upgrade --project tests`, and
`tests/test_conventions.py` is what reports whether anything JAX promises moved.

### How the oracle suite is built, and why it matters when editing it

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is too much text about the oracle test suite. Let's omit a lot of this or make it shorter. It's distracting for the purposes of the contrib guide.

Comment thread CONTRIBUTING.md Outdated

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can get rid of this whole section (enumerating the checks). This is pretty self evident, no?

Comment thread CONTRIBUTING.md Outdated
— see the comments in
[`.github/workflows/release.yml`](.github/workflows/release.yml), including the
optional upgrade to crates.io Trusted Publishing.
- PyPI uses **Trusted Publishing (OIDC)**, so there is no token to leak or

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can omit this cause it is only needed to be done once, and it is done.

Comment thread README.md Outdated
## Status

**M2 landed and released.** The scalar engine, the DataFusion adapter and the
Python wheel are all published; ddx runs on DataFusion and DuckDB, and its

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wait, we don't have DuckDB support yet.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's at least say "eventually".

- "load-bearing" -> "necessary" for `--reinstall-package ddxdb`.
- Cut the oracle-suite explanation down to the two things that would
  otherwise make a failure confusing, and point at tests/README.md for
  the rest. The design rationale belongs there, not in a guide someone
  reads to get set up.
- Dropped the enumerated CI checks. The names and their output are on
  every pull request already, so the list was a second copy that could
  only go stale.
- Dropped the one-time PyPI pending-publisher instructions, which have
  been carried out.

README: the DuckDB claim was too strong. "Runs on DataFusion and DuckDB"
reads as two engines with equal support, when only DataFusion has a
native adapter — bare grad() in ordinary SQL and through the DataFrame
API. DuckDB works through `rewrite_sql` like every other engine: rewrite
the text, hand it to your own connection. The status section now draws
that line and says the in-database extension comes eventually (M5),
which is also what the table's `ddx-duckdb` row has said all along.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

@alxmrs alxmrs left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM for now.

@alxmrs
alxmrs merged commit 776b35b into main Aug 9, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant