Styxis
Company
Knowledge

Checking Iceberg delete positions before trusting a head limit

A delete at position 100 cannot address a four-row file; validate the metadata before checking output size.

Approve positions before scan limits

Validate delete positions separately from the final row limit. Counting an out-of-range position as a real deletion can affect how a reader compensates for deleted rows, even though no row was removed. Put separate checks on the incoming metadata and the scan output. [1, 2, 3]

Five requested rows, six returned

The Polars 2.1.0 reproduction associates delete position 100 with a four-row file. Collecting first and applying head(5) returns five rows, while pushing head(5) into the lazy scan returns six in the report. This observed path difference does not establish corruption in every Iceberg file. [1]

Use the boundary of the referenced file

Join each delete position to its file's row count and quarantine invalid positions before scanning. Independently check the head result's upper bound. Positions start at 0 and must be non-negative and smaller than that file's row count. A total across several files is the wrong boundary for a position in one file. [1, 3]

An upper-bound example and a separate lower bound

For an illustrative four-row file, position 3 is the last valid position; 100 fails the requirement to be less than 4. These two cases demonstrate the upper bound. A separate lower-bound condition must reject negative positions. If the file length is unknown, obtain its metadata rather than replacing the boundary with a generous estimate. [1, 3]

The scoped Truthound check

Truthound Core's ExpressionValidator can apply both boundaries to prepared file_rows and delete_position fields. This is an input check that isolates invalid deletion metadata. Compare requested and returned row counts afterward as a separate outcome check. The predicate neither establishes the accuracy of the supplied file length nor repairs Polars' deletion-and-limit execution order. [1, 2, 3]

Application conditions

Prepare a position audit containing file_id, file_rows and delete_position from original metadata or an independent control, aligned to the same run, file and key scope.

Alternatives

Quarantine invalid position-delete metadata and read only files whose metadata passes the input policy.

Limitations

  • The file row count must itself be accurate. Truthound does not repair the deletion/limit order in an Iceberg scan.

Evidence

Event checked
Subject checked
Modified