Styxis
Company
Knowledge

Iceberg nested equality deletes: verify survivors as well as deletion targets

Preservation of nontarget rows is a separate acceptance condition, not an optional consequence of deleting the intended keys.

Deletion has two acceptance conditions

Check both absence of deletion targets and preservation of nontargets. Supplying only a deleted-row count to Truthound can make excessive deletion look successful. Start with an independently prepared survivor-key set. [1, 2, 3]

Spark and generic readers disagree

The public Iceberg reproduction uses an equality-delete key nested in a struct. The Spark reader removes every row, while the core generic reader returns the expected survivors. Preserve the nested-key condition and reader path; the report does not establish that every equality delete is affected. [1]

Choose the surviving keys first

Calculate deletion targets and survivors independently, then compare survivor sets across readers. Do not approve deletion by its total count alone. Preserve parent identities and NULL policy when flattening nested paths, and never derive the expected set from the same faulty reader. [1, 3]

Count is only an initial signal

If an illustrative approved survivor set contains 5760 rows, observing 5760 satisfies a necessary count condition and 0 fails. A different set of 5760 rows must still fail the final check. Derive the expected count from the independently defined survivor set. [1, 3]

Compare the exact set with Truthound

Truthound Core's DistinctValuesEqualSetValidator can compare observed survivor keys with the approved set. Repeated business keys require a separate row identity or multiplicity check. This does not repair Spark's nested-key projection. When readers disagree, withhold downstream use of the affected result and preserve both the original data and delete files. [1, 2, 3]

Application conditions

Independently derive row_id, nested_key and should_survive from the source for the same deletion scope, linking them to the actual survivor-key set.

Alternatives

Compare survivor sets from the core generic reader and Spark reader.

Limitations

  • Truthound does not repair Spark nested-key projection. Deriving the expected set with the same faulty reader invalidates the comparison.

Evidence

Event checked
Subject checked
Modified