MAINT: New declarative mini-DSL and port of readers - #441
Merged
Conversation
…zation Expression conversion hooks in BinaryArray.read were called without the parser context: inspect.signature raises on operator-overloaded expression objects (its internals compare the object against type/object, which builds an expression whose truth value is an error), and the resulting fallback silently dropped the context. This broke data reading in the AL3D and ZON readers, whose conversions reference the parsed header. Expressions are now dispatched explicitly, here and at the other hook call sites (XML converters, channel bindings, mask rules). Further fixes from the review of the declarative-reader rework: - ValidationError is now a subclass of CorruptFile (and thus ReadFileError), matching the corrupt_file error taxonomy of the format description contract - isnan(null) evaluates to true, as the contract specifies - The exporter refuses to rewrite native byte order @ to = when a field uses a native-size code (l L n N P) whose width would change - Nested layout objects in a BinaryStructure field list serialize to opaque markers instead of crashing the encoder, and refuse to rehydrate - Unset hooks are detected via a shared _identity default instead of fragile __defaults__ introspection - Channel-binding info entries that evaluate to None are omitted (matching the historic readers); raw_metadata is kept verbatim - Seek and Switch gained the same to_dict convenience as If and Skip - The exporter's reader/fixture registry is the single source of truth for the round-trip gate tests Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JpBHNcgJUbo7XD8gHvPkYV
…ders All four are now lambda-free, declare magic-byte detection and channel bindings, and are enrolled in the exporter registry, extending the round-trip CI gate and the exported format description documents from seven to eleven formats. Supporting DSL changes: - TLVContainer passes previously parsed named entries into the context of subsequent entries' layouts (like within a CompoundLayout), so the FRT topography block can take its array shape from the grid block parsed earlier in the tag list - TLVContainer gained a 'default' layout for tags missing from the tag map (e.g. Skip), so unknown blocks can be skipped lazily instead of being read into memory - New registry function 'omit' (mapping without a key), used to keep the lazy data handle out of FRT's raw_metadata - make_datetime accepts an optional seventh argument, a UTC offset in minutes, producing timezone-aware timestamps (used by VK) Behavior notes: the OS3D, NMS and FRT plausibility checks now raise reader exceptions (FileFormatMismatch/CorruptFile) instead of plain ValueError/AssertionError; NMS keeps its file-size check (its de-facto detection) as Python-side validation; raw_metadata of the four readers is now keyed by the parsed structures rather than the ad-hoc keys of the old implementations. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JpBHNcgJUbo7XD8gHvPkYV
…ders All four are now lambda-free and enrolled in the exporter registry, extending the round-trip CI gate and the exported format description documents from eleven to fifteen formats. Supporting DSL changes, gated behind a new 'text' capability: - New layout node TextHeader: line-oriented key-value headers with a configurable separator or fixed-width key column, comment prefixes, INI-style or key-triggered sections, per-key converter expressions, and several delimiting modes (byte count, terminator line, stop key, end of stream) - New layout node TextLine: a single stripped text line - New layout node TextMatrix: whitespace-separated number matrices with bad-value markers parsed as NaN - New core layout node Check: validates a condition against the context without reading from the stream (cross-structure validation) - New registry functions: split, flip, mangle_length_unit, is_length_unit - Undefined-data mask rules now receive the parsed metadata as their context, so a rule can depend on header fields (e.g. SDF's data-type-specific invalid marker) Robustness fixes uncovered while porting: - isin() encodes its values individually, so byte-string alternatives (e.g. BCR's UTF-16 byte-order marks) serialize correctly - Literals now refuse to serialize values that would not survive a JSON round trip (e.g. mappings with integer keys), failing at export time instead of producing documents that silently misbehave The DI reader remains imperative for now: its soft/hard-scale reconciliation, dynamic cross-section lookups and tuple-unit non-height channels are procedural logic that would not gain clarity from the declarative form. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JpBHNcgJUbo7XD8gHvPkYV
All three are now lambda-free and enrolled in the exporter registry, extending the round-trip CI gate and the exported format description documents from fifteen to eighteen formats. Supporting DSL changes: - New layout node ForEach: repeats a structure for each element of a previously parsed list (e.g. the payloads of OPD's block directory), with the current element and its index in the context - New layout node Let: stores computed values into the context, e.g. to make the item of an enclosing ForEach part of a structure's result - ZipContainer member names can be expressions over previously parsed members (data files named within an XML index, as in PLUX and X3P), a member entry can loop over a list (one member per element, e.g. per measurement layer), stream-less nodes (Check, Let) can be interleaved between members, and a mismatch_error parameter selects the exception for a stream that is not a ZIP archive or a missing required member (FileFormatMismatch when the member identifies the format) - New registry functions: pluck, values_with_prefix, reshape, isfinite, logical_not, unpackbits (ISO 5436-2 per-point validity masks) Behavior notes: OPD now declares magic-byte detection (the 'Directory' block name at offset 2) instead of answering 'maybe' for every file; X3P channel metadata now includes the parsed main.xml as raw_metadata (the historic reader assembled a raw-metadata dictionary but never reported it); the up-front archive-member size checks of PLUX and X3P are dropped and mismatches surface when the data is read. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JpBHNcgJUbo7XD8gHvPkYV
All three are now lambda-free and enrolled in the exporter registry, extending the round-trip CI gate and the exported format description documents from eighteen to twenty-one formats. Supporting DSL changes: - New layout node TIFFContainer (capability 'tiff'): reports the pages of a TIFF file, each with its tag values (standard names, names from a private-tag dictionary, or hexadecimal codes; enums as [name, value] pairs; byte blobs and arrays omitted from metadata), a lazy handle to the page raster with an optional conversion expression, repeating private-tag blocks collected into record lists (JPK's data-interpretation slots), and tag layouts that parse the byte value of a private tag with ordinary layout nodes (Park Systems' vendor header and data tags), with lazy readers re-reading the tag on demand - Channel bindings gained 'where' (an expression filtering which foreach elements emit a channel, e.g. JPK's length-unit channels) and 'checks' (per-channel validations raising a taxonomy error, e.g. JPK's scaling-type constraints) - New registry functions: match_records, match_records_containing (record selection by key path, e.g. LEXT's HEIGHT page), index_by, and parse_xml (vendor XML embedded in TIFF tags) - The error taxonomy name-to-class map moved to Exceptions.ERROR_CLASSES so bindings and the serializer share one definition Behavior notes: the readers answer 'maybe' for any TIFF file and 'no' otherwise (as before for PS; LEXT and JPK previously answered 'maybe' for every file); LEXT's raw_metadata now reports the raw tag values per page plus the parsed measurement and device-settings XML instead of the historic mixed representation, and its DateTime tag is reported verbatim rather than ISO-normalized. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JpBHNcgJUbo7XD8gHvPkYV
pastewka
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.