Do not assume local dates are monotone across a midnight clock fallback
Local dates need not increase with UTC through a midnight fallback. Fix the historical timezone interval and expression type when comparing pruned key sets, and do not treat disabling skipping as a cure for every rewrite.
Later instants can have earlier dates
A timestamp can increase while its local date moves backward after midnight. Min/max date transformation is then an unsafe pruning assumption. Truthound Core can compare selected keys with an independently established local-date reference. [1, 2, 3]
The historical St_Johns interval
Hudi issue #20525 reports master with Spark 3.5 pruning historical files in America/St_Johns on 2009-11-01. 02:30:30Z renders as November 1, while later 02:40Z renders as October 31. The report separates file-statistics pruning from a Spark CAST rewrite with a similar assumption. [1]
Separate expression controls
Use the same file with skipping enabled and disabled across the historical fallback. Record UTC, zone, local date, file bounds, and result keys. Keep date_format and CAST controls separate: disabling skipping does not correct every expression rewrite. [1, 3]
Key 2 belongs to October 31
Key 2 at UTC 2009-11-01 02:40:00 has local date 2009-10-31 in the declared St_Johns reference. The date_format predicate expects set 2. Returning 2 passes; empty fails because the file excluded a matching historical row. [1, 3]
Timezone history is external
DistinctValuesEqualSetValidator in Truthound Core Python SDK 3.3.2 compares the extracted key column with expected_values. The operator establishes timezone history and the reference. The SDK does not modify tzdb or Hudi’s matcher. [1, 2, 3]
Modern DST is insufficient coverage
Validate a UTC-range or explicit-date path appropriate to the query, maintaining separate date_format and CAST tests. A contemporary daylight-saving example may never exercise the narrow historical midnight reversal. [1, 3]
One consistent tzdb can be nonmonotone
Keep the historical time interval in the acceptance artifact. The problem is not disagreement between two timezone databases; it is a nonmonotone local-date transformation even under one consistent database. [1, 3]
Application conditions
Prepare UTC, historical zone/local date, and skipping settings for the same file.
Alternatives
Separate date_format/CAST controls and verify a UTC-range alternative.
Limitations
Disabling skipping does not also correct a separate Spark CAST rewrite.
Evidence
- Event checked
- Subject checked
- Modified
