Styxis
Company
Knowledge

Keep nested delete ordering values in Hudi event-time reconciliation

Replay a late delete against a newer update for the same key. Record the nested ordering path and table version, then check the survivor against the event-time policy.

Deletes also carry event order

A delete must preserve its original event order. A default value produced by a missing nested-field lookup is not a valid replacement timestamp. Truthound Core can compare the final survivor state after the operator prepares an ordered event reference. [1, 2, 3]

A nested ordering report

Hudi issue #20367 reports stale deletes removing newer rows in Spark MOR table versions 8 and 9 when nested ordering fields are written with default values. Follow-up discussion explains that nested ordering is an existing Spark-side configuration, distinguishing this silent path from a separate version 10 writer exception. [1]

Compare events for the same key

Replay a newer update and an older delete for the same key in controlled nested-field and top-level-field paths. Record record-merger type, table version, original ordering values, and final key version. Use the actual EVENT_TIME_ORDERING policy rather than borrowing commit-time expectations. [1, 3]

The 100 versus 50 fixture

Define id1 at meta.ts=100 and a delete at meta.ts=50. Event-time ordering leaves version 100 alive; that passes. Represent absence only in the inspection table with the declared marker 0. A final marker 0 fails because the older delete removed the newer row. [1, 3]

Keep presence separate from version

DistinctValuesEqualSetValidator in Truthound Core Python SDK 3.3.2 can compare the prepared survivor-version values with an approved set. Keep key presence alongside the version marker so a real version 0 cannot be confused with absence. The SDK does not rewrite delete blocks or infer nested paths. [1, 2, 3]

Test the same ordering policy

Compare a top-level ordering field or the reported Avro record path in the actual environment before choosing a supported route. These alternatives still require the same event-order policy; changing to commit-time ordering changes the contract. [1, 3]

Another surviving key is insufficient

Do not treat another key surviving as evidence that this key retained its newer event. The meaningful check joins the old delete and newer update for the same identity, preserving both order and the final presence decision. [1, 3]

Application conditions

Prepare update/delete ordering, merger type, and table version for the same key.

Alternatives

Compare top-level or Avro paths under the same event-order policy.

Limitations

  • Do not apply EVENT_TIME_ORDERING expectations unchanged to commit-time policy.

Evidence

Event checked
Subject checked
Modified