Styxis
Company
Knowledge

Declare Parquet statistic order alongside Trino min and max

Before consuming Parquet min/max, check each leaf's column_orders declaration against schema order. Passing the declaration check still leaves the numerical correctness of the bounds to independent validation.

Values need declared ordering

Statistics need a declared comparison order as well as values. Truthound Core can inspect a prepared footer table before another reader is expected to use those bounds for pruning. [1, 2, 3]

A footer declaration report

Trino issue #31946 reports its writer emitting min_value and max_value without FileMetaData.column_orders. Readers relying on a default may use them, while the specified order is undefined. A follow-up links a PR adding TYPE_ORDER entries; deployment remains unverified here. [1]

Map entries in leaf order

Extract leaf order, dtype, min/max presence, and column_orders entries from each footer. Require an interpretable declaration for every leaf whose statistics are used. Preserve schema order rather than checking only the total number of entries. [1, 3]

The two-leaf prerequisite

Define an illustrative file with 2 statistical leaves. Both need corresponding order entries; prepared complete=true passes. false fails and blocks statistics-pruning approval. That Boolean is a presence condition, not a claim that the bounds themselves are accurate. [1, 3]

Checking does not rewrite footers

ColumnPairValidator in Truthound Core Python SDK 3.3.2 can compare prepared non-null declaration flags with true. The operator reads the footer and defines supported order semantics. The SDK does not rewrite Parquet metadata or define order for unsupported types. [1, 2, 3]

Verify consuming-reader behavior

Verify the writer footer and the consuming reader’s actual statistics behavior together. Withhold pruning based on undefined-order bounds instead of relying silently on one library’s default. [1, 3]

Declaration and accuracy are distinct

Keep declaration and numeric accuracy as separate obligations. A valid-looking min/max pair cannot communicate its ordering meaning by itself across implementations. [1, 3]

Application conditions

Extract leaf order, dtype, statistics presence, and column_orders from footers.

Alternatives

Verify writer footers and the actual consuming reader’s use of statistics together.

Limitations

  • Declaration presence does not prove bound accuracy or every reader’s use.

Evidence

Event checked
Subject checked
Modified