Do not read truncated Delta timestamp statistics as exact upper bounds
Preserve the precision difference between millisecond statistics and microsecond values. Read duplicate reporting and the follow-up fix together.
A boundary between different precisions
Truncated timestamp statistics are not exact bounds on original values. Admit pruning only after preserving the precision contract and reconciling expected row IDs. A truncated maximum below the actual value cannot justify skipping the matching record; hand off the statistic’s unit with its conservative interpretation. Derive expected row IDs supplied to Truthound from raw timestamp filtering that retains full precision. [1, 2, 3]
A duplicate with a reported 2.1.0 fix
Polars #30180 reports sub-millisecond matches missing when a Delta timestamp maximum truncated to milliseconds is treated as exact. A follow-up identifies duplicate #30147 and says #30194 widens the bound through the millisecond end, released in 2.1.0. That attributed release statement is not this article’s execution of the reader’s installation. [1]
Derive expected IDs from raw timestamps
Preserve Delta-statistic units, truncation policy, original timestamps, and a fixed snapshot. Derive expected IDs from original values and hold scans that omit them. Retain full precision in an independent reference and represent absent output explicitly as missing. [1, 3]
Compare 123456 with 123400
The illustrative timestamps have fractional seconds .100000 and .123456, with a predicate greater than .123400. Only id=2 satisfies the raw comparison, so expected ID 2 is independent of scan output. Actual 2 passes; missing fails. Check additions and duplicates separately. [1, 3]
The row-presence comparison boundary
ColumnPairValidator in Truthound Python SDK 3.3.2 compares expected_row_id derived from original values by the operator with normalized actual_row_id. Operators interpret file statistics and execute reads. Core does not repair pruning or statistic interpretation, and one matching ID does not establish the complete result. [1, 2, 3]
Costs of pruning alternatives
Compare verified nonpruning reads or a path using conservative statistics, measuring the cost. Do not count the duplicate report as another underlying event. Existing same-boundary regressions reduce the need for another audit, while each actual installation still needs its own support check. [1, 3]
Application conditions
Retain raw timestamp precision, the Delta snapshot, statistic units, and independent expected IDs.
Alternatives
Compare verified nonpruning or conservative-statistics paths alongside their costs.
Limitations
One matching ID does not establish every row, duplicate, or payload, or repair file pruning.
Evidence
- Event checked
- Subject checked
- Modified
