Fix #906: execute viral propagation only at combination points (main, DuckDB-only)#908
Draft
javihern98 wants to merge 6 commits into
Draft
Fix #906: execute viral propagation only at combination points (main, DuckDB-only)#908javihern98 wants to merge 6 commits into
javihern98 wants to merge 6 commits into
Conversation
DuckDB-only adaptation of the 1.9.X change (cr-906 / PR #907). Viral propagation runs and is required only where input data points are combined (>=2 datasets, aggregation/analytic group-by, hierarchy roll-up); everywhere else viral attributes are copied unchanged. Adds require_rules/ combined_viral_components, moves the 1-3-3-6 check from the global per-statement site into the combining operators' validate methods, and makes the row-preserving DuckDB emissions copy (removing vp_dataset_wide_sql). nvl copies its primary operand. Supersedes #897's dataset-wide check_datapoint viral aggregate (a row-preserving validation now copies per datapoint). Fixes #906.
…uired) (cr-906-main)
…ne in group/partition (cr-906-main)
… in DuckDB (cr-906-main)
…h 1.9.X (cr-906-main)
…ropagation, joins, keep, parsing, validation) (cr-906-main)
|
|
||
| # -- check_datapoint validation operator -- | ||
|
|
||
| _DPR = """ |
|
|
||
| # -- Rule execution on row-preserving dataset-level operators (issue #877) -- | ||
|
|
||
| _AGG_RULE = "define viral propagation VP (variable VAt_1) is aggregate max end viral propagation;" |
| # -- Rule execution on row-preserving dataset-level operators (issue #877) -- | ||
|
|
||
| _AGG_RULE = "define viral propagation VP (variable VAt_1) is aggregate max end viral propagation;" | ||
| _ENUM_RULE = ( |
| {"name": "VAt_1", "type": "Number", "role": "Viral Attribute", "nullable": True}, | ||
| ], | ||
| } | ||
| _GP_STR_DS = { |
| ], | ||
| } | ||
|
|
||
| _ENUM_PAIR_RULE = """ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
DuckDB-only adaptation for
mainof the 1.9.X change in #907 (issue #906). Viral attribute propagation was executed (and required) on essentially every dataset operator; per the VTL 2.2 Attribute propagation rule it must run only when input data points are combined, and otherwise copy viral attributes unchanged.if/casewith ≥2 dataset branches); an aggregation/analytic group-by/partition; thehierarchyroll-up.calc/keep/drop/rename/unpivot,cast, time shifts,nvl, set operators,check_datapoint/check_hierarchy.The
SemanticError 1-3-3-6moves from a global per-statement check into the combining operators'validatemethods (still fires duringsemantic_analysis()), flagging only the viral attributes actually combined. The row-preserving DuckDB SQL emissions become plain copies (vp_dataset_wide_sqlremoved);nvlcopies its primary operand (COALESCE-filled).mainis DuckDB-only, so there is no pandasevaluate/apply_row_preservingpath (this is the difference from the 1.9.X PR #907).Fixes #906.
Checklist
ruff format,ruff check,mypy)pytest) — full suite: 5056 passed, 11 skipped1-3-3-6description;error_messages.rstregenerates frommessages.py)Impact / Risk
abs(DS_1)), and an enumerated rule no longer remaps values — both now copy. A viral attribute never combined no longer requires a rule.check_datapointaggregate the viral attribute over the whole validation result; as a row-preserving validation it now copies the viral value per datapoint. The13-1expected output is updated accordingly. Please confirm this is the intended direction.Notes
nvlis single-operand for propagation.main.pyworking-tree change is excluded.