Styxis
Company
Knowledge

Project non-null union branches when validating the Hudi file-group reader

Reconcile file-group projection of an Avro union with the legacy path using branch values and member representations. Success for a top-level name cannot approve an untested nested branch.

Project the union itself

Reading an unrelated name column does not prove a union column is readable. Exercise each non-null branch and projection. Truthound Core can compare prepared branch-aware outcomes with an independently declared representation. [1, 2, 3]

A rewrite after Parquet reading

Hudi issue #20528 reports Hive/MR file-group reads on 1.3.0-SNAPSHOT failing while rewriting a Parquet member group to an Avro UNION target. Legacy reading returns member structs. The report distinguishes the successful underlying Parquet read from the later schema rewrite. [1]

Control each non-null branch

For COPY_ON_WRITE, keep one non-null fixture per union branch and compare file-group and legacy projections over the same file. Preserve branch, value, read status, and member structure instead of accepting a name-only control. [1, 3]

The member0 reference

For illustrative union [null,int,string] with value 1, the approved meta-sync representation is member0=1,member1=null. Tag MEMBER0_1 passes; READ_ERROR fails. A missing output must not be replaced with an invented member struct. [1, 3]

Decode union meaning externally

ColumnPairValidator in Truthound Core Python SDK 3.3.2 compares prepared non-null representation tags. The operator decodes union branches and supplies the reference. The SDK does not interpret Avro unions or implement Hudi’s schema rewrite. [1, 2, 3]

Use the same projection

Verify the legacy-reader control using the same file and projection before choosing a supported reader. A successful branch does not automatically cover record unions or arrays containing unions. [1, 3]

Keep partial compatibility visible

Store projection and branch alongside the result. This keeps partial compatibility visible and prevents a successful top-level field from approving a nested path that was never exercised. [1, 3]

Application conditions

Prepare each non-null union-branch projection from the same COW file.

Alternatives

Verify the legacy reader with the same projection before selecting a supported path.

Limitations

  • A name-only success does not establish union, record-union, or union-array support.

Evidence

Event checked
Subject checked
Modified