bench: external-engine A/B benchmark harness + pair corpus (first external engine: DuckDB) — waves A–D - #1570
Draft
aaj3f wants to merge 8 commits into
Draft
bench: external-engine A/B benchmark harness + pair corpus (first external engine: DuckDB) — waves A–D#1570aaj3f wants to merge 8 commits into
aaj3f wants to merge 8 commits into
Conversation
…, protocol Ports the v0.3 paired A/B harness into a committable, sanitized bench-duckdb-ab/ tree: - harness/ (run_pair.py, summarize_ab.py, sanitized targets.json template, pairs/) - substrate/ (MinIO + Iceberg REST fixture compose, pyiceberg loaders incl. partitioned probe, runbook) - PROTOCOL.md (binding rules: memo-COLD/data-WARM fresh-process primary, RSS disclosed not caps-equal, full-result-consumption boundary, substrate field discipline, SQL<->SPARQL equivalence, DNF caps) - RESULTS.md (wave tables; substrate-B four-way baseline seeded, p1/p2 flagged load-contaminated pending Wave A) Sanitized: no account IDs, hostnames, secrets, absolute/session paths.
Re-ran p1/p2 under clean load (load1 7.7-9.0) on substrate B: DuckDB + fluree-main + fluree-#1528, both cache modes, N=5. Replaces the load-contaminated baseline rows. Cold: DuckDB wins p2 ~1.8x, near-tie p1; Warm: Fluree wins p1 (count-manifest shortcut) and p2 ~2x. main==#1528 on p1/p2 (no regression).
Wrote FACT_WEB_EVENT (1M rows) to local MinIO partitioned 252 ways (identity month-bucket + genuine iceberg month-transform). DuckDB probes: grouped-agg scan OK sub-second at default AND threads=1; fact-dim join with join_filter_pushdown ON succeeds with NO bloom error on both layouts + force-on. Verdicts: Finding 1 = Snowflake-managed-S3-specific (not many-files); Finding 2 = NOT tied to partitioned multi-file layout (corrects PREP §10) = Snowflake-managed-substrate-specific. load_tables_partitioned.py made self-contained (identity on derived bucket, no pyiceberg_core needed). Posted to issue #1568.
aaj3f
changed the title
…ousands-comma parse fix
Adds cq038/cq014/cq027/cq008/cq040/cq016/cq046 (corpus-based) + crt_join_reorder/crt_minmax/crt_highcard_groupby (RT additions) with per-pair equivalence rules in manifest.json. check_pair.py cross-checks SQL==SPARQL by NAME-aligned columns (exact/tolerance/rows_only, IRI->key normalization) before timing. run_pair.py fix: CLI prints thousands-separator commas at >=1000 rows ('(1,100 rows, ...)'), which broke the tally regex; now accepts commas. All 10 pairs pass correctness.
Renames bench-duckdb-ab/ -> bench-external-ab/ and restructures the harness so external engines are adapters (harness/engines/fluree.py, engines/duckdb.py, engines/common.py); run_pair.py + check_pair.py are engine-agnostic cores that dispatch to the adapters. Adds the verbatim third-party NOTICE (nominative use; no DuckDB code/binaries/platform incorporated, external CLI invoked as a documented user install) at the top of README.md and PROTOCOL.md. Results language made factual (measurements, not competitive framing); where Fluree lags is stated plainly. Sanitization re-verified: only the <SNOWFLAKE_ACCOUNT> placeholder host remains.
engines/common.py: time_argv() picks /usr/bin/time -v on Linux (GNU, RSS in KB) vs -l on macOS (BSD, bytes); parse_rss() normalizes both to bytes. Adapters use common.time_argv() instead of a hardcoded -l. Enables running the same harness on a Linux benchmark host (arch_prefix set empty there).
…e arm64 linux) Full uncontested table (host-identity stamped: c7g.4xlarge/AL2023, duckdb 1.5.5 d8cdaa33fd sha 9882c99a, fluree 5598ffd + c81862d), predictions-vs-outcomes vs R1 §3.7, and named-gaps. Findings: #1528 fusion collapses cq038 (74s->99ms, ~750x) and cq016 (30s->1.0s, ~30x) beyond its scoped p3 case; crt_join_reorder (multi-FACT join) is the largest still-open gap (~130s, ~900x, unaffected by #1528); crt_highcard 14x + 8x RSS; fused/relational pairs 3-6x. Local contended legs demoted to labeled secondary. Correctness gate green all 10.
Full pair set at 10x scale, same uncontested host. Scale-delta: the fused/scan-heavy family's 3-6x tail widened to 6-52x (fluree per-row materialization scales linearly vs DuckDB sub-linear); the multi-FACT join (crt_join_reorder) went from 917x-completes to DNF (>200s) on both binaries (DuckDB 321ms) — a hard wall at scale; #1528 fusion becomes essential (cq038 ~100ms vs main 72s; cq016 completes ~22s vs main DNF); fluree RSS widened (crt_highcard 4.4GB vs DuckDB 512MB). 153 rows accounted for (scoped legs to bound instance-hours; n/a vs DNF distinguished).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
DRAFT — the maintainer flips this after review.
A reproducible A/B benchmark that uses DuckDB as a single-node yardstick for Fluree's virtual (R2RML-over-Iceberg) query path over the SF0.1
ENTERPRISE_DEMOBI star schema. Thesis under test: Fluree should at least match DuckDB per comparable query on the same physical Iceberg files; every DuckDB win beyond noise is a named engine gap with a query shape attached.Everything lands under one new top-level directory,
bench-duckdb-ab/.What's here
PROTOCOL.md— the binding rules: memo-COLD/data-WARM fresh-process-per-rep primary (no engine result-memo can serve a timed run), peak RSS disclosed rather than caps equalized, timing ends at full result consumption, per-row substrate/version discipline, correctness-first, SQL↔SPARQL equivalence rules, DNF caps, machine discipline.harness/—run_pair.py(paired runner, protocol-compliant),summarize_ab.py, a sanitizedtargets.jsontemplate (substrate-A Snowflake configs are<placeholder>templates only; the OAuth secret is read from$SF_SECRETat runtime and never stored), andpairs/(the SQL↔SPARQL pair corpus +manifest.jsonwith per-pair equivalence rules).substrate/—docker-compose.yml(MinIO +apache/iceberg-rest-fixture),load_tables.py(unpartitioned) +load_tables_partitioned.py(Wave-B probe), and a runbookREADME.md.RESULTS.md— the wave tables.Substrate
Fluree reads only catalog-fronted Iceberg, so both engines read a catalog-fronted table. Substrate A (Snowflake-managed, deployment-real) carries catalog/manifest ops; substrate B (local MinIO + REST fixture, engine-pure) carries the data-scanning headline numbers. DuckDB is pinned ≥ 1.5.3 (runs use v1.5.5,
arch -arm64). Two provenance-stamped Fluree binaries: shipped-main and a build carrying the #1528 filter-over-join fusion.Waves (filled incrementally in
RESULTS.md)Sanitization: no account identifiers, hostnames, secrets, or absolute/session paths in any committed file.