Project non-null union branches when validating the Hudi file-group reader
Reconcile file-group projection of an Avro union with the legacy path using branch values and member representations. Success for a top-level name cannot approve an untested nested branch.
Project the union itself
Reading an unrelated name column does not prove a union column is readable. Exercise each non-null branch and projection. Truthound Core can compare prepared branch-aware outcomes with an independently declared representation. [1, 2, 3]
A rewrite after Parquet reading
Hudi issue #20528 reports Hive/MR file-group reads on 1.3.0-SNAPSHOT failing while rewriting a Parquet member group to an Avro UNION target. Legacy reading returns member structs. The report distinguishes the successful underlying Parquet read from the later schema rewrite. [1]
Control each non-null branch
For COPY_ON_WRITE, keep one non-null fixture per union branch and compare file-group and legacy projections over the same file. Preserve branch, value, read status, and member structure instead of accepting a name-only control. [1, 3]
The member0 reference
For illustrative union [null,int,string] with value 1, the approved meta-sync representation is member0=1,member1=null. Tag MEMBER0_1 passes; READ_ERROR fails. A missing output must not be replaced with an invented member struct. [1, 3]
Decode union meaning externally
ColumnPairValidator in Truthound Core Python SDK 3.3.2 compares prepared non-null representation tags. The operator decodes union branches and supplies the reference. The SDK does not interpret Avro unions or implement Hudi’s schema rewrite. [1, 2, 3]
Use the same projection
Verify the legacy-reader control using the same file and projection before choosing a supported reader. A successful branch does not automatically cover record unions or arrays containing unions. [1, 3]
Keep partial compatibility visible
Store projection and branch alongside the result. This keeps partial compatibility visible and prevents a successful top-level field from approving a nested path that was never exercised. [1, 3]
Application conditions
Prepare each non-null union-branch projection from the same COW file.
Alternatives
Verify the legacy reader with the same projection before selecting a supported path.
Limitations
A name-only success does not establish union, record-union, or union-array support.
Evidence
- Event checked
- Subject checked
- Modified
