Feat/arrow testing integration - #185
Draft
maltzsama wants to merge 7 commits into
Draft
Conversation
Add `apache/arrow-testing` as a Git submodule to provide a shared IPC conformance corpus used across all Arrow implementations. - Add `testing` submodule pointing to arrow-testing - Add `ArrowTestData` helper to locate corpus files via ARROW_TEST_DATA env var or fallback to submodule path - Add test that verifies path resolution - Exclude .gitmodules from RAT check This is the first step toward replacing the Go-based data generator with the official test corpus, enabling broader conformance coverage.
Adds conformance tests reading generated_primitive.stream and its no-batches and zero-length variants from the arrow-testing corpus. The existing streaming tests write with ArrowWriter.writeStreaming and read the result back with ArrowReader.readStreaming, which establishes internal consistency but not conformance: a symmetric defect in the writer and reader would pass. Until now no test read a stream produced by another Arrow implementation. The shared schema expectations are lifted out of the file conformance tests so both classes use them. Note: testStreamAndFileAgree is disabled pending a fix to ArrowBuffer memory initialization (see inline comment for details); it will serve as the regression test when that issue is resolved. Part of apache#10
When running from a git clone (dev or CI), initialize the arrow-testing submodule. When running from an extracted tarball (RC verification), clone the repository at build time instead, since submodules are not present in the archive produced by git archive. Adds git config --global --add safe.directory, since the copied working tree in the Docker build has different ownership than the process running git, which git rejects by default.
These files are produced by data-generator/swift-datagen at build time (see ci/scripts/build.sh) and should not appear as untracked noise in git status.
Open
Reads generated_duplicate_fieldnames.arrow_file from the arrow-testing corpus, which carries a struct column alongside two top-level fields sharing the same name. Neither is covered by the existing conformance tests, which only read flat primitive types. Only schema and shape are asserted. Values are not, because fields with no validity buffer currently read uninitialized memory and are not reproducible between runs. Part of apache#10
testFileWriter_bool and testFileWriter_struct exercise ArrowWriter; the fixture files were only a source of input data. They now build their record batches with the array builders, following the pattern already used by makeRecordBatch and makeStructRecordBatch in the same file. Removes testFileReader_bool, testFileReader_double and testFileReader_struct. Their coverage is now provided by IPCFileConformanceTests, which reads equivalent data produced by another Arrow implementation rather than by the local generator, across more types and more rows. No test depends on data-generator/swift-datagen after this change. Part of apache#10
data-generator/swift-datagen produced three Arrow files consumed by the file reader tests. Those tests now either read gold files from the arrow-testing corpus or build their data with the Swift array builders, so the generator has no remaining consumers. Removes the generator, its build step in ci/scripts/build.sh, the Go toolchain from ci/docker/ubuntu.dockerfile, its dependabot entry, the now-obsolete rat_exclude_files.txt rule for go.sum under two levels, the .gitignore rules for the files it produced, and the corresponding section of Sources/Arrow/README.md. swift test now requires only a Swift toolchain and the arrow-testing data, which ci/scripts/build.sh fetches when the submodule is not initialized. Verified end-to-end via docker compose build && docker compose run: 52 tests pass, 0 failures, with no Go toolchain in the image. CDataWGo is unaffected: it tests the C Data Interface against a real Go runtime and cannot be replaced by static test data. Part of apache#10
Contributor
Author
|
If you can take a look on that and give me some orientation I will be very happy to apply any modification needed |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft: IPC Conformance Testing via arrow-testing
Adds apache/arrow-testing submodule and conformance tests for file and stream
readers. Currently includes:
In progress:
Related to #10.