Skip to content

fix(validation): reject duplicate mapping keys - #337

Open
iamrajatrana wants to merge 2 commits into
apache:mainfrom
iamrajatrana:fix/validator-duplicate-keys
Open

fix(validation): reject duplicate mapping keys#337
iamrajatrana wants to merge 2 commits into
apache:mainfrom
iamrajatrana:fix/validator-duplicate-keys

Conversation

@iamrajatrana

@iamrajatrana iamrajatrana commented Aug 19, 2026

Copy link
Copy Markdown

Summary

Reject duplicate keys in YAML and JSON mappings before schema validation can run against silently overwritten data. The loader remains derived from yaml.SafeLoader, adds no dependency, preserves aliases and legal merge-key overrides, and reports both mapping and duplicate-key source locations.

This PR also adds Validation CI so the validator tests and canonical example run on Python 3.11 through 3.14 whenever validation, core specification, or canonical example files change.

Before

source: staging.orders
source: production.orders
$ uv run validation/validate.py duplicate.yaml
Validation PASSED: duplicate.yaml
$ echo $?
0

After

$ uv run validation/validate.py duplicate.yaml
Error: Invalid YAML: while constructing a mapping
...
found duplicate key 'source'
...
$ echo $?
1

Edge cases covered

  • Top-level and nested duplicate keys
  • Quoted and explicitly tagged equivalent keys
  • Duplicate JSON object keys
  • Duplicate scalar keys whose values are collections
  • A valid duplicate hiding an earlier invalid value
  • Explicit duplicates alongside YAML merge keys
  • Repeated merge keys
  • Same key in separate mappings remains valid
  • YAML aliases remain valid
  • Legal merge-key overrides remain valid
  • Merge keys remain distinct from a quoted literal "<<" key
  • Error marks identify the original mapping and duplicate location
  • End-to-end non-zero exit for duplicates and successful validation for valid input

Merge-key behavior

A mapping may contain one YAML merge key and explicitly override values inherited through that merge; this remains supported. Repeating the special << merge key in the same mapping is rejected because YAML provides the sequence form (<<: [*first, *second]) for merging multiple mappings and mapping keys are expected to be unique.

Scope notes

  • This change is intentionally limited to validation/validate.py. Several converters load their own vendor-specific YAML with yaml.safe_load; applying a shared strict-loading policy across those independent packages requires separate compatibility evaluation.
  • Empty documents and top-level sequences can still reach the validator's pre-existing data.get(...) assumption and raise AttributeError. That behavior is unrelated to duplicate-key detection and is not changed here.
  • Tests are invoked as uv run validation/test_validate.py, matching their PEP 723 dependency metadata and script-local import path.

Related Issues

Closes #336

Tests

  • uv run validation/test_validate.py (16 tests)
  • uv run validation/validate.py examples/tpcds_semantic_model.yaml
  • Duplicate-key integration matrix across YAML and JSON inputs
  • uv run --with ruff ruff check --ignore EXE001 validation/validate.py validation/test_validate.py
  • Validation CI matrix: Python 3.11, 3.12, 3.13, and 3.14

Checklist

  • New behavior is covered by unit and integration tests
  • Validation tests are enforced in GitHub Actions
  • Existing canonical semantic model validation still passes
  • ASF license headers are present on new files
  • No third-party dependency was added

Signed-off-by: iamrajatrana <rjtrana16@gmail.com>
Signed-off-by: iamrajatrana <rjtrana16@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

validator: duplicate YAML and JSON keys are silently overwritten

1 participant