Styxis
Company
Knowledge

Iceberg upserts after partition evolution: check updates and deletes of existing keys

Successful writes to a new partition spec do not establish that rows written under the old spec were removed.

The change reaches beyond file layout

Partition evolution also changes the boundary over which existing keys must be updated. Giving Truthound only newly written files hides matching keys still present in old files. Inspect the complete current state being approved. [1, 2, 3]

Changes that miss older rows

The public Iceberg report describes Flink upsert and CDC changes failing to remove rows written before a partition-spec change. Old and new versions can coexist, or a deleted key can remain visible. Compaction having run is not sufficient evidence that this state was corrected. [1]

Include pre-transition keys in controls

Choose keys written before the change, then test an update and a delete after the spec transition. Check duplicate live versions and post-delete survival separately. Testing only newly inserted keys misses the cross-spec boundary. Record original keys, versions, old and new specs and the intended final operation. [1, 3]

Update and delete have different final states

In an illustrative completed update, 1 live version per key is normal and 2 fails. A completed delete instead requires that key to be absent. Uniqueness alone is insufficient: a failed delete leaving only an old version can still appear unique. [1, 3]

Apply Truthound to the current state

Use Truthound Core's UniqueValidator to inspect current business-key duplication, then separately check whether deletion targets survive. Obtain intended final operations from CDC records. Truthound does not fix equality-delete applicability. Any chosen repair needs the same controls, including keys from the earlier spec, to validate its result. [1, 2, 3]

Application conditions

Link spec_transition, business_key, row_version and deletion_expectation to original CDC records and the complete current state.

Alternatives

Validate a separate consistency-repair operation on keys spanning the partition-spec transition.

Limitations

  • Absence of duplicate keys does not establish successful deletion; also check tombstone targets. Truthound does not repair equality-delete applicability.

Evidence

Event checked
Subject checked
Modified