Styxis
Company
Knowledge

Match Hudi incremental pruning statistics to historical file coverage

A current snapshot can conceal pruning omissions in a historical incremental read. Fix the requested timeline range and reconcile its keys with a control read covering that same range.

Historical coverage is the prerequisite

Statistics are safe for an incremental read only when they cover its selected historical files. Newer statistics are not automatically better. Truthound Core can compare historical matching keys with a separately established expected set. [1, 2, 3]

Current statistics versus historical matches

Hudi issue #20480 describes Flink incremental reads on 1.3.0-SNAPSHOT excluding historical matches after a partition value changes from age 23 to 50. Follow-up discussion says an end boundary at the latest commit is not sufficient by itself: start boundaries, commit limits, and skipped operations also affect coverage. [1]

Fix the selected timeline

Fix start and end completion times and the selected file range. Compare key sets with partition-statistics pruning disabled for that same range. Record read.commits.limit and relevant skip settings; a current snapshot cannot supply the historical answer. [1, 3]

The age 23 reference set

At commit 1, id1 and id2 both have age=23. Later only id1 becomes 50. For the declared commit 1 range, the expected matching set is id1|id2. Returning both passes. Returning only id2 applies later state to historical evidence and fails. [1, 3]

Operator-prepared historical keys

DistinctValuesEqualSetValidator in Truthound Core Python SDK 3.3.2 can compare the exported key column with that expected set. The operator must establish timeline coverage and normalize keys. The SDK does not reconstruct a Hudi timeline or restore partition statistics. [1, 2, 3]

Separate pruning mechanisms

Separate partition-predicate pruning from statistics on nonpartition values. The follow-up favors conservatively disabling partition-statistics pruning for incremental reads until safe cases are established, while other pruning mechanisms may remain available. This is a discussed response, not a measured universal performance prescription. [1, 3]

Retain the requested past

The acceptance question is whether the requested past keys survive, not whether current statistics appear plausible. Keep the range and reference extraction with the result so future readers can distinguish a genuine historical absence from pruning loss. [1, 3]

Application conditions

Fix start/end completion times, selected files, and skip settings.

Alternatives

Withhold incremental partition-stat pruning until its coverage is established.

Limitations

  • Do not replace the historical expected set with a current snapshot.

Evidence

Event checked
Subject checked
Modified