Validate NULL array elements across Hudi list-encoding paths
Arrays containing NULL elements need a round-trip check through each consumer path after the write succeeds. Track parent validity, element positions, and NULL state separately to expose empty-array substitution or lost elements.
Nullability crosses reader boundaries
Array element nullability is a round-trip contract across writers and readers, not just a write option. Truthound Core can compare a prepared element-position NULL bitmap after the operator verifies the selected file encoding. [1, 2, 3]
The scope of one setting experiment
Hudi issue #20505 reports Spark SQL INSERT with NULL array elements failing under the default Parquet Avro list encoding in 1.2.0. A driver-wide setting succeeded in the reported experiment, while other writers, readers, compaction, and clustering were not validated. [1]
Separate arrays and parent validity
Separate ordinary arrays, arrays with NULL elements, empty arrays, and NULL parents. For the chosen encoding, match parent keys, positions, and NULL states after reading through each real consumer. A single successful INSERT does not cover the whole exchange path. [1, 3]
Preserve the second element
Fix the illustrative source array at [1,NULL,3]. At the second position, expected is_null=true. true passes that position’s missingness condition when the parent and all three positions remain. false fails because replacing the element with a number changes the array meaning. [1, 3]
Align positions before comparison
ColumnPairValidator in Truthound Core Python SDK 3.3.2 can compare non-null position-level is_null flags against the source bitmap. The operator supplies parent/position alignment. The SDK does not convert Parquet list encoding or infer missing array positions. [1, 2, 3]
Cover consuming and rewriting paths
Test the reported setting only in a controlled environment first, then collect evidence from every actual consuming and rewriting path before switching. Keep the original encoding and data available for comparison. [1, 3]
An empty array is different data
An empty array cannot serve as a successful substitute for [1,NULL,3]. Counts, parent validity, and element validity are separate observations, and changing one to make a writer accept the input alters the contract. [1, 3]
Application conditions
Align parent keys, element positions, and NULL bitmaps across writer and actual readers.
Alternatives
Verify the encoding in a controlled environment, then cover every consuming path.
Limitations
One driver-setting success does not guarantee compaction or other-reader compatibility.
Evidence
- Event checked
- Subject checked
- Modified
