Checking required children only where their parent struct exists
Physical NULL slots beneath an absent parent are different from missing children under a present parent.
Define where requiredness applies
Evaluate a nested child's requiredness on rows where its parent actually exists. Physical slots beneath a NULL nullable struct are not present child values. A present parent with a missing required child is a real violation. A single unqualified null_count conflates those situations. [1, 2, 3]
A writer rejection that ignores the parent mask
The Arrow report describes a Parquet writer checking physical NULLs in a required leaf without accounting for the parent struct's validity bitmap. The reproduction has physical NULLs in both levels while full Table validation succeeds. It illustrates disagreement between nested requiredness and the writer's handling path. [1]
Extract both validity states
Extract parent and child validity together and check required-child missingness only under present parents. Join parent_present and required_child_present by row identity. Filling masked slots with zero or an empty string merely to pass a writer can make an absent child appear to be real data. [1, 3]
A masked NULL versus a genuine violation
Define an illustrative normal dataset with masked NULLs only beneath absent parents and all required children populated beneath present parents. Its violation count is 0. Add one present parent with an absent required child and the count becomes 1, which fails. Adding another NULL-parent row should not increase that count. [1, 3]
Conditional validation and writer selection
Truthound Core's ConditionalExpressionValidator can require child presence only when parent_present is true. Prepare validity extraction externally. The check does not directly remove the Parquet writer's rejection, so verify parent-state preservation when choosing a supported writer or an alternative representation. [1, 2, 3]
Application conditions
Prepare a validity audit of parent_present and required_child_present from original data or an independent control, aligned to the same run, file and key scope.
Alternatives
Validate a supported writer or alternate representation that preserves parent validity.
Limitations
The check does not directly resolve Parquet writer rejection. Flattening or choosing another writer requires a separate check of representation preservation.
Evidence
- Event checked
- Subject checked
- Modified
