fix(metadata): fix ArrayIndexOutOfBoundsException when 1.x reader merges old MDT records - #19560
fix(metadata): fix ArrayIndexOutOfBoundsException when 1.x reader merges old MDT records#19560lokeshj1703 wants to merge 1 commit into
Conversation
…ges old MDT records When a 1.x reader reads MDT data written by an older Hudi release, log blocks carry the old HoodieMetadataRecord writer schema (6 fields, no SecondaryIndexMetadata). HoodieMetadataPayload.getInsertValue() uses an object-identity check against the current HOODIE_METADATA_AVRO_SCHEMA (7 fields). The old schema fails this check and falls into the else-branch, which assumes HOODIE_META_COLUMNS are prepended and uses TYPE_FIELD_OFFSET=6 — causing ArrayIndexOutOfBoundsException on a 6-field schema. Fix: add isHoodieMetadataRecordSchema() that recognises any version of HoodieMetadataRecord by Avro name + namespace, routing it to the fast path. Old schemas produce a current-schema record with SecondaryIndexMetadata=null. Closes apache#19559 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
hudi-agent
left a comment
hudi-agent
left a comment
There was a problem hiding this comment.
Thanks for working on this! The PR fixes an ArrayIndexOutOfBoundsException when a 1.x reader merges MDT records written by an older Hudi release with a narrower HoodieMetadataRecord schema, by broadening the fast-path check in getInsertValue to also accept any HoodieMetadataRecord-named schema. The core fix looks reasonable and the returned record's schema is self-consistent with what the merger propagates downstream. One edge case around the breadth of the schema match is worth double-checking in the inline comment. Please take a look, and this should be ready for a Hudi committer or PMC member to take it from here. Code looks clean overall — one minor naming nit below.
| // Uses cached Avro schema reference for O(1) equality check. | ||
| if (schema == null || schema == HOODIE_METADATA_AVRO_SCHEMA) { | ||
| // If the schema is same or none is provided, we can return the record directly | ||
| if (schema == null || schema == HOODIE_METADATA_AVRO_SCHEMA || isHoodieMetadataRecordSchema(schema)) { |
There was a problem hiding this comment.
🤖 isHoodieMetadataRecordSchema only checks the record name/namespace, so it matches any schema named HoodieMetadataRecord — including one with the 5 Hudi meta columns prepended (addMetadataFields preserves the name) or a projected subset. Those cases previously fell into the else-branch. Now they take the fast path and get back the full 7-field record regardless of the requested layout. Is that safe for every caller here (e.g. a populateMetaFields=true MDT or a projected read), or could it hand back a record that doesn't match the schema the caller asked for?
| return MetadataPartitionType.get(type).combineMetadataPayloads(previousRecord, this); | ||
| } | ||
|
|
||
| private static boolean isHoodieMetadataRecordSchema(Schema schema) { |
There was a problem hiding this comment.
🤖 nit: isHoodieMetadataRecordSchema describes the structural check but not the intent — could you rename it to something like isCompatibleMetadataRecordSchema or isOlderVersionMetadataRecordSchema to signal that this exists specifically to handle older schemas missing newer fields?
3 participants
Describe the issue this Pull Request addresses
Closes #19559
When a Hudi 1.x reader reads a Metadata Table (MDT) that was written by an older Hudi release, merging log-file delta records against base-file records throws an
ArrayIndexOutOfBoundsException:Summary and Changelog
Root cause:
SecondaryIndexMetadatawas added toHoodieMetadataRecordin a later Hudi release, growing the schema from 6 to 7 fields. MDT log blocks written by older releases carry the 6-field writer schema in their block header. When merging,HoodieAvroRecordMergerretrieves the writer schema from the buffered record and passes it toHoodieMetadataPayload.getInsertValue(schema).Inside
getInsertValue, the fast path uses an object-identity check:The old 6-field schema is a different
Schemaobject fromHOODIE_METADATA_AVRO_SCHEMA(7 fields), so it fails the identity check and falls into theelsebranch designed for schemas withHOODIE_META_COLUMNSprepended. Writing to index 6 on a 6-field schema throwsArrayIndexOutOfBoundsException.Fix: Add
isHoodieMetadataRecordSchema(Schema)that recognises any version ofHoodieMetadataRecordby Avro record name and namespace. Route these older-version schemas to the same fast path as the current schema, returning aHoodieMetadataRecordwith the current schema (SecondaryIndexMetadatadefaults tonull).Changes:
HoodieMetadataPayload.java: addisHoodieMetadataRecordSchema()helper; extend the fast-path guard ingetInsertValue()TestHoodieMetadataPayload.java: addtestGetInsertValueWithOldMetadataSchemaregression test that constructs the old 6-field schema and verifies no exception is thrownImpact
No user-facing API or config change. Tables written by older Hudi releases can be read safely by a 1.x reader without requiring a table upgrade.
Risk Level
Low. The change is confined to
getInsertValue()inHoodieMetadataPayload. The added condition only triggers when the schema is aHoodieMetadataRecordschema that is not object-identical toHOODIE_METADATA_AVRO_SCHEMA(i.e. an older-version schema read from a log block); all other callers are unaffected.Documentation Update
None.
Contributor's checklist