Styxis
Company
Knowledge

Test savepoint reads after Hudi cleanup, beyond file retention

Exercise the specified commit's AS OF read after cleaning, even when savepoint files remain. Keep retained inventory separate from consumer-read status to distinguish absent files from a rejected read boundary.

Retention and readability differ

Retained recovery files and an allowed timestamp query are separate conditions. Truthound Core can compare a prepared post-clean read state with a reference captured at the savepoint before cleanup. [1, 2, 3]

A retained c2 with a rejected read

Hudi issue #20482 reports 1.2.1 with Spark 3.5.5 retaining a c2 savepoint file but rejecting TIMESTAMP AS OF c2 through earliestCommitToRetain validation. The same query returns the savepoint row before cleaning. File existence alone therefore does not establish readable recovery. [1]

Keep the exact instant

Connect savepoint commit, retained files, clean watermark, AS OF status, and expected row. Do not declare recovery ready from inventory alone. Preserve the exact requested instant rather than substituting a newer readable snapshot. [1, 3]

The pre-clean ROW_1_B reference

At c2 the source independently records (id=1,v=b). Use illustrative state ROW_1_B after cleaning. ROW_1_B passes; RETENTION_ERROR fails even though the file remains. The tag records both the selected recovery point and its content. [1, 3]

Inspect the actual read result

ColumnPairValidator in Truthound Core Python SDK 3.3.2 compares prepared non-null read-state tags. The operator executes the read and preserves its earlier reference. The SDK does not bypass time-travel validation or restore a savepoint. [1, 2, 3]

Verify the recovery promise

Validate cleanup policy and the supported recovery procedure in an isolated environment. Check a real savepoint read before relying on that retained point in an operational recovery promise. [1, 3]

A readiness gap is not physical erasure

A file-retention test and a consumer-read test can disagree without contradiction. Recording both gives the operator a precise readiness gap instead of an unsupported claim that cleanup physically erased the savepoint. [1, 3]

Application conditions

Prepare the pre-clean c2 reference row and later file/watermark/AS OF states.

Alternatives

Verify supported clean/savepoint recovery procedures in isolation.

Limitations

  • Retained files do not guarantee reader selection or successful reads at that instant.

Evidence

Event checked
Subject checked
Modified