Styxis
Company
Knowledge

Checking required children only where their parent struct exists

Physical NULL slots beneath an absent parent are different from missing children under a present parent.

Define where requiredness applies

Evaluate a nested child's requiredness on rows where its parent actually exists. Physical slots beneath a NULL nullable struct are not present child values. A present parent with a missing required child is a real violation. A single unqualified null_count conflates those situations. [1, 2, 3]

A writer rejection that ignores the parent mask

The Arrow report describes a Parquet writer checking physical NULLs in a required leaf without accounting for the parent struct's validity bitmap. The reproduction has physical NULLs in both levels while full Table validation succeeds. It illustrates disagreement between nested requiredness and the writer's handling path. [1]

Extract both validity states

Extract parent and child validity together and check required-child missingness only under present parents. Join parent_present and required_child_present by row identity. Filling masked slots with zero or an empty string merely to pass a writer can make an absent child appear to be real data. [1, 3]

A masked NULL versus a genuine violation

Define an illustrative normal dataset with masked NULLs only beneath absent parents and all required children populated beneath present parents. Its violation count is 0. Add one present parent with an absent required child and the count becomes 1, which fails. Adding another NULL-parent row should not increase that count. [1, 3]

Conditional validation and writer selection

Truthound Core's ConditionalExpressionValidator can require child presence only when parent_present is true. Prepare validity extraction externally. The check does not directly remove the Parquet writer's rejection, so verify parent-state preservation when choosing a supported writer or an alternative representation. [1, 2, 3]

Application conditions

Prepare a validity audit of parent_present and required_child_present from original data or an independent control, aligned to the same run, file and key scope.

Alternatives

Validate a supported writer or alternate representation that preserves parent validity.

Limitations

  • The check does not directly resolve Parquet writer rejection. Flattening or choosing another writer requires a separate check of representation preservation.

Evidence

Event checked
Subject checked
Modified