Styxis
Company
Knowledge

Validate NULL array elements across Hudi list-encoding paths

Arrays containing NULL elements need a round-trip check through each consumer path after the write succeeds. Track parent validity, element positions, and NULL state separately to expose empty-array substitution or lost elements.

Nullability crosses reader boundaries

Array element nullability is a round-trip contract across writers and readers, not just a write option. Truthound Core can compare a prepared element-position NULL bitmap after the operator verifies the selected file encoding. [1, 2, 3]

The scope of one setting experiment

Hudi issue #20505 reports Spark SQL INSERT with NULL array elements failing under the default Parquet Avro list encoding in 1.2.0. A driver-wide setting succeeded in the reported experiment, while other writers, readers, compaction, and clustering were not validated. [1]

Separate arrays and parent validity

Separate ordinary arrays, arrays with NULL elements, empty arrays, and NULL parents. For the chosen encoding, match parent keys, positions, and NULL states after reading through each real consumer. A single successful INSERT does not cover the whole exchange path. [1, 3]

Preserve the second element

Fix the illustrative source array at [1,NULL,3]. At the second position, expected is_null=true. true passes that position’s missingness condition when the parent and all three positions remain. false fails because replacing the element with a number changes the array meaning. [1, 3]

Align positions before comparison

ColumnPairValidator in Truthound Core Python SDK 3.3.2 can compare non-null position-level is_null flags against the source bitmap. The operator supplies parent/position alignment. The SDK does not convert Parquet list encoding or infer missing array positions. [1, 2, 3]

Cover consuming and rewriting paths

Test the reported setting only in a controlled environment first, then collect evidence from every actual consuming and rewriting path before switching. Keep the original encoding and data available for comparison. [1, 3]

An empty array is different data

An empty array cannot serve as a successful substitute for [1,NULL,3]. Counts, parent validity, and element validity are separate observations, and changing one to make a writer accept the input alters the contract. [1, 3]

Application conditions

Align parent keys, element positions, and NULL bitmaps across writer and actual readers.

Alternatives

Verify the encoding in a controlled environment, then cover every consuming path.

Limitations

  • One driver-setting success does not guarantee compaction or other-reader compatibility.

Evidence

Event checked
Subject checked
Modified