Styxis
Company
Knowledge

Classify missing Hudi clean instants by the sync window

Classify an absent archived clean by whether it belongs to the window required for the current sync. If required evidence is missing, reconcile a verified full partition inventory at the same boundary before declaring completion.

Different absences need different policy

Missing old history and missing input required by the current sync need different error policies. Truthound Core can inspect a prepared required-window classification without treating every archived clean instant as permanently necessary. [1, 2, 3]

A sync and archiver report

Hudi issue #20379 reports meta sync racing with the archiver while reading cleans already covered by earlier syncs. It distinguishes those obsolete records from missing cleans inside the needed window. A follow-up links a correction PR but does not establish the installed version. [1]

Keep listing and opening times

Fix last synced commit and the clean window needed for the next partition update. Record list and open times plus instant identity. If required evidence is missing, obtain a verified full-partition-list comparison rather than silently declaring the sync complete. [1, 3]

One currently required clean

Designate one illustrative clean after last sync as necessary to explain a current partition deletion. Its expected availability=true. true passes this input prerequisite. false fails if ignored like an old 404, because the current deletion calculation then lacks its required evidence. [1, 3]

Classification belongs to the operator

ConditionalExpressionValidator in Truthound Core Python SDK 3.3.2 can require availability for rows classified as currently required. The operator defines that classification and collects the records. The SDK does not stop archiving or synchronize Hive or Glue. [1, 2, 3]

Align the fallback boundary

Validate a supported fallback comparing a full partition list and catalog list at the same boundary. The shape of a wrapped S3 exception cannot identify whether its deletion was already applied. [1, 3]

Permanent history is not implied

Retain enough provenance to explain why an absent instant was irrelevant or essential. Permanent retention of all old instants is not implied by this contract; correct classification of the current input window is. [1, 3]

Application conditions

Fix last sync and the clean window required for the current deletion calculation.

Alternatives

Verify a full-partition-list fallback at the same boundary.

Limitations

  • Do not infer prior application from exception wrapping alone.

Evidence

Event checked
Subject checked
Modified