witchtools 0.6.0: fast region-conversion engine + japan/korea/southafrica region sets - #28
Merged
Merged
Conversation
Move the iso3-explosion conversion core of convert_region() verbatim to convert_region_via_iso3() in R/convert_region_legacy.R. convert_region() keeps validation, mapping resolution and the pass-through shortcut, then delegates. No behaviour change; this isolates the reference engine so a faster engine can be added and tested for equivalence against it.
with_engine() option helper, a GLOBIOM-shaped input generator covering the edge cases (NA values, per-id missing regions, duplicated rows, factor and character-period columns), and expect_region_equal() asserting identical names/classes and tolerance-compared values. Smoke-tested legacy-vs-legacy until the fast engine lands.
Region-to-region conversion no longer explodes the table to iso3 country level (~x15 rows for witch17 input, held across four full-size intermediates). The downscale->upscale is a linear map, so the fast engine precomputes a per-(from, to) pair coefficient table (21 rows for witch17->witch20) and converts with one small join (~x1.2 rows) and one grouped aggregation. Covers sum, sumby (including the info coperture), mean, min/minw and max/maxw for all agg_missing modes; column and row order match the legacy engine exactly. iso3-level input (already linear-size) and the set1 operator stay on the legacy engine: set1's round() is discontinuous, so the fast engine's summation reassociation could flip values sitting exactly on a .5 boundary (observed on synthetic data). The legacy engine remains selectable via options(witchtools.convert_region_engine = "legacy") and serves as the reference in the new equivalence suite (127 assertions across operators, agg_missing modes, routes, weights, NA patterns, per-id missing regions, duplicated rows, non-partition mappings and row order).
subset() on a data.table or character vector allocates through the data-frame path; plain [ filtering is faster and copies less. Four call sites: the index-column selection in convert_gdx/convert_duckdb/ convert_sqlite and the missing-period filter in convert_time_period.
benchmarks/run_all.sh runs each cell in a fresh Rscript subprocess under
a systemd scope (MemoryMax, swap disabled) so over-budget runs are
OOM-killed and recorded as DNF instead of swap-thrashing the host; peak
RSS comes from GNU time. Matrix: synthetic GLOBIOM-shaped tables (1e5,
1e6, 4e6 rows) x {sum, mean, sumby} witch17->witch20, a witch17
pass-through, iso3->witch17, and two real 2.14M-row tables from the
witch-master GLOBIOM DuckDB, each x {legacy, fast}.
Results (6G cap, committed in benchmarks/results/results.md): the
legacy engine is OOM-killed on every 4e6-row case and on BOTH real
GLOBIOM tables, while the fast engine converts them in ~6s at <1GB
peak. Where both complete: 4.5-5.5x faster, 3.2-9.7x lower peak RSS,
identical checksums. Pass-through and iso3 routes unchanged (1.0x).
Coefficient-table build is ~14ms, so caching it across the convert_gdx
item loop is not worth the added state (decision recorded here).
Also declare the sw_from NSE symbol flagged by R CMD check.
Emit is_japan, is_korea and is_southafrica in the generated regions.inc, alongside the existing is_* sets. Needed by WITCH bunker modules to set region-specific bounds that stay valid across region mappings (jpnkor under witch17, japan/korea under witch20). Selection uses the same GDP-majority rule as the other region sets, so under witch17 jpnkor belongs to is_japan and is_korea is empty.
witch_write_gams() now emits is_japan, is_korea and is_southafrica, so regions.inc declares 17 sets, not 14. Also check is_japan membership against japan_regions().
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Release candidate for 0.6.0 (minor: new features, backward compatible).
Performance — fast region-conversion engine
convert_region()(andconvert_table()/convert_gdx()/convert_duckdb()/convert_sqlite()through it) no longer expands data to country level: a precomputed region-pair coefficient table drives the conversion.benchmarks/results/).options(witchtools.convert_region_engine = "legacy"); iso3 input and theset1operator always use the legacy path.test-convert-region-equivalence.R) covers NAs, per-id missing regions, duplicated rows, non-partition mappings, and row order.New features — region sets
japan_regions(),korea_regions(),southafrica_regions(), emitted asis_japan/is_korea/is_southafricain the generatedregions.inc(GDP-majority rule; under witch17,jpnkormaps tois_japanandis_koreais empty). Needed by the WITCH bunker modules for region-specific bounds valid across mappings.witch_write_gams()test now expects the 17 emitted sets and checksis_japanmembership.Release
DESCRIPTIONbumped to 0.6.0, NEWS finalised.R CMD check: OK (0 errors, 0 warnings, 0 notes).To be merged manually after CI is green on all five platforms.
🤖 Generated with Claude Code