Styxis
Company
Knowledge

NULLs after a DataFusion Parquet Variant merge: check presence by layout

Distinguish a field missing in the source from a reader-created NULL caused by representation handling.

Merging files with different representations

Investigate a NULL increase after schema merging by separating source absence from conversion loss. One file may store a field in a typed area while another retains it in original value bytes. A whole-table completeness ratio supplied to Truthound can hide losses concentrated in one layout. [1, 2, 3]

Struct padding conflicts with Variant meaning

The public DataFusion report describes mixed Parquet Variant shredding layouts. Ordinary struct NULL-padding does not preserve the meaning of residual Variant value bytes, so a merged typed field can appear NULL despite a value remaining in the source. This is different from a writer losing the source field itself. [1]

Preserve evidence of source presence

Compare original and decoded field presence by file layout. Hold approval of the combined result if a field disappears only in one layout. Prepare file identity, layout kind, row identity and original presence together. Do not derive expected presence using the same problematic merged reader. [1, 3]

A layout-specific counterexample

Define an illustrative source with the field present in 6 rows. It should remain present in 6 decoded rows. If one layout loses values and the count drops to 3, the condition fails. This is not a rule to fill genuinely missing source values. Apply it only to rows with evidence that the source field exists. [1, 3]

The conditional Truthound check

Truthound Core's ConditionalExpressionValidator can require decoded presence when the original presence flag is true. An independent reader that understands the Variant bytes must extract those states first. Truthound does not directly decode Variant or infer every layout. Reading each layout with a verified reader before explicit merging is an alternative, but it still requires checks of values and types. [1, 2, 3]

Application conditions

Link file_layout, source_field_present and decoded_field_present for the same file and row. Establish source presence with an independent Variant reader.

Alternatives

Decode each Variant layout with a supporting reader, then merge explicitly.

Limitations

  • Truthound neither decodes Variant bytes directly nor infers shredding rules automatically. An independent reader must supply source-field presence.

Evidence

Event checked
Subject checked
Modified