[HUDI-5621] Decouple Hive from integ-test-bundle [FaaFyfxR9WAQrL7FcAgEHJvztd8cVMxvjHRS55rw1nwH] - #19580
Conversation
Remove the direct org.apache.hive:* dependencies from the integ-test bundle pom, along with the corresponding shade includes and relocations. The Hive functionality is still available through the separate hudi-hive-sync-bundle module, which is the dedicated bundle for Hive sync operations. This prevents the integ-test bundle from forcing Hive on all users and follows the same pattern as HUDI-9020 (Remove HBase dependencies). Fixes apache#15724
hudi-agent
left a comment
hudi-agent
left a comment
There was a problem hiding this comment.
Thanks for working on this! This PR removes the direct org.apache.hive:* dependencies, shade includes, and relocations from the integ-test bundle to decouple Hive. One thing worth double-checking before merge: the shade config is an allow-list, and the bundle still ships hudi-hive-sync and hudi-integ-test main code that references Hive classes at runtime (embedded HiveServer2, Hive sync DAG nodes) — see the inline comment on whether those classes are still available on the runtime classpath. Please take a look at the inline comment, and this should be ready for a Hudi committer or PMC member to take it from here.
| <include>org.apache.hive:hive-exec</include> | ||
|
|
||
|
|
||
| <include>com.yammer.metrics:metrics-core</include> |
There was a problem hiding this comment.
🤖 This shade <artifactSet><includes> is an allow-list, so dropping the four org.apache.hive:* includes means those classes are no longer packaged into the bundle even though they still resolve transitively via hudi-hive-sync. But the bundle still includes hudi-hive-sync (line 84) and hudi-integ-test (line 88), and their main code references Hive at runtime — e.g. HiveServiceProvider imports org.apache.hive.service.server.HiveServer2 (hive-service), and HiveSyncNode/HiveSyncTool pull in hive-jdbc/metastore/common. Won't the test-suite DAGs that run Hive sync / start the embedded HiveServer2 from this jar now hit NoClassDefFoundError, unless Hive is guaranteed on the runtime classpath externally? Could you confirm how the integ-test-bundle is launched (is Hive provided by the container/spark classpath), or whether these Hive includes still need to stay?
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #19580 +/- ##
============================================
- Coverage 77.49% 77.49% -0.01%
+ Complexity 32784 32781 -3
============================================
Files 2522 2522
Lines 139140 139140
Branches 16723 16723
============================================
- Hits 107832 107822 -10
- Misses 23742 23748 +6
- Partials 7566 7570 +4
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
|
Thank you for the contribution, can you please read through the contribution guide and adhere to the PR template? On top of that, please refrain from including your Solana wallet address in the title or any of the comments here. We do not endorse/encourage soliciting of payments when contributing to OSS. If this is not done in the next 24 hours, we will be closing all your PRs with such signatures in order to not set a dangerous precedence. Thank you. |
5 participants
Change Logs
Remove the direct
org.apache.hive:*dependencies from the integ-test bundle pom, along with the corresponding shade includes and relocations. The Hive functionality is still available through the separatehudi-hive-sync-bundlemodule, which is the dedicated bundle for Hive sync operations.Impact
hudi-integ-test-bundleno longer bundles Hive artifacts directlyhudi-hive-sync-bundlehudi-hive-sync(Hudi's own module) is still included in the bundle as beforeChecklist
[HUDI-5621] ...packaging/hudi-integ-test-bundle/pom.xmlorg.apache.hive:*dependencies (hive-exec, hive-metastore, hive-jdbc, hive-common)org.apache.hive:*andorg.apache.hadoop.hive:*shade includesFixes #15724