Styxis
Company
Knowledge

Trino filters after a Hive decimal change: inspect Parquet storage type and scale

Interpreting old integer-file statistics through only the current schema can prune matching rows.

Logical types and physical statistics can diverge

After a logical-type change, compare how physical statistics are interpreted as well. A Truthound set comparison can reveal disagreement between read paths, but explaining it requires both the original Parquet storage type and the current decimal scale. [1, 2, 3]

Full reads work while filtering loses matches

The public Trino reproduction uses older integer Parquet files under a Hive column changed to a scaled decimal. Unfiltered reading works, while statistics-based filtering misses matching rows; a statistics-ignoring path is the control. This is different from concluding that all historical files are physically damaged. [1]

Use an independent full read as the reference

Record storage type, current logical type and scale. Compare pruning results with keys obtained by fully decoding and then filtering in the same logical units. If cost limits the test to selected files, retain that scope in the report instead of implying complete coverage. [1, 3]

An illustrative small match set

If the independent full read establishes 1 matching key, the filtered path must return that same 1. Returning 0 fails. Returning another single key has the same count but still fails set equality. Obtaining the reference first through the statistics-based path would make the check circular. [1, 3]

Compare with Truthound and retain the limits

Truthound Core's DistinctValuesEqualSetValidator can compare approved matching keys with the pushdown output. It does not rewrite Parquet statistics. Use a verified full-read-then-filter alternative or retest the same historical files after correcting type and statistic handling. Do not extend the conclusion automatically beyond the files inspected. [1, 2, 3]

Application conditions

Preserve physical_type, logical_scale, predicate and matching_row_id for the same historical files, deriving expected keys through independent full decoding.

Alternatives

Obtain filtered results through a full-read control without statistics pruning.

Limitations

  • Truthound does not rewrite Parquet statistics. Full-read controls have a cost, so specify the inspected files and scope.

Evidence

Event checked
Subject checked
Modified