Why unknown NaN-page statistics cannot be omitted from row-group bounds
An unknown range is different from an empty page. Check that the advertised lower bound does not exclude original valid values.
Missing contribution versus unknown range
An unknown page range is not an empty contribution. Verify that row-group statistics conservatively cover valid original values before using them for pruning. Valid statistics from another page cannot bound the unknown page merely because their numbers look precise. Prepare written statistics and independent raw-valid-value ranges in a table for a Truthound admission check. [1, 2, 3]
Separate the AI-attributed diagnosis
Polars #29640 reports a NaN-containing Parquet page dropping out of row-group bounds, with an actual 100.0 row missing from a scan filter. The report attributes its causal analysis to AI. A follow-up proposes changes along that diagnosis; this article does not independently execute the writer or measure a patch. [1]
Retain page and row-group boundaries
Retain valid raw values, the NaN mask, page and row-group boundaries, and written statistics. Hold pruning based on an advertised minimum greater than the original minimum, then compare an independent read. Distinguish absent page statistics from an empty page and use a separate test destination. [1, 3]
A lower bound that must include 0
The illustrative row group has page A=[0,NaN,100] and page B=[200]. The valid overall minimum is independently 0. Advertised minimum 0 passes; 116508 fails because it excludes original 0 and 100. Upper bounds, NaN and NULL policies, and the actual row set still need separate checks. [1, 3]
The statistics-audit boundary
ColumnPairValidator in Truthound Python SDK 3.3.2 compares advertised_minimum extracted by the operator with independent_valid_minimum. Raw reading, page interpretation, and range calculation are prepared externally. Core does not repair the Parquet writer or pruning code, and one lower-bound match does not prove full coverage. [1, 2, 3]
Nonpruning reads and row-ID reconciliation
Compare a verified nonpruning reader and writer settings, reconciling expected row IDs across the complete result. Existing independent range checks can make another audit redundant. Do not substitute total row count for matching the actual retained observations. [1, 3]
Application conditions
Externally distinguish valid raw values, NaN masks, page boundaries, and absent statistics when deriving the reference range.
Alternatives
Compare verified nonpruning reads and writer settings, reconciling expected row IDs too.
Limitations
Passing the lower bound does not establish upper bounds, NULL and NaN policies, or full row preservation.
Evidence
- Event checked
- Subject checked
- Modified
