NULLs after a DataFusion Parquet Variant merge: check presence by layout
Distinguish a field missing in the source from a reader-created NULL caused by representation handling.
Merging files with different representations
Investigate a NULL increase after schema merging by separating source absence from conversion loss. One file may store a field in a typed area while another retains it in original value bytes. A whole-table completeness ratio supplied to Truthound can hide losses concentrated in one layout. [1, 2, 3]
Struct padding conflicts with Variant meaning
The public DataFusion report describes mixed Parquet Variant shredding layouts. Ordinary struct NULL-padding does not preserve the meaning of residual Variant value bytes, so a merged typed field can appear NULL despite a value remaining in the source. This is different from a writer losing the source field itself. [1]
Preserve evidence of source presence
Compare original and decoded field presence by file layout. Hold approval of the combined result if a field disappears only in one layout. Prepare file identity, layout kind, row identity and original presence together. Do not derive expected presence using the same problematic merged reader. [1, 3]
A layout-specific counterexample
Define an illustrative source with the field present in 6 rows. It should remain present in 6 decoded rows. If one layout loses values and the count drops to 3, the condition fails. This is not a rule to fill genuinely missing source values. Apply it only to rows with evidence that the source field exists. [1, 3]
The conditional Truthound check
Truthound Core's ConditionalExpressionValidator can require decoded presence when the original presence flag is true. An independent reader that understands the Variant bytes must extract those states first. Truthound does not directly decode Variant or infer every layout. Reading each layout with a verified reader before explicit merging is an alternative, but it still requires checks of values and types. [1, 2, 3]
Application conditions
Link file_layout, source_field_present and decoded_field_present for the same file and row. Establish source presence with an independent Variant reader.
Alternatives
Decode each Variant layout with a supporting reader, then merge explicitly.
Limitations
Truthound neither decodes Variant bytes directly nor infers shredding rules automatically. An independent reader must supply source-field presence.
Evidence
- Event checked
- Subject checked
- Modified
