DuckDB ENUM Parquet distinct_count: allowed values are not observed values
The number of declared ENUM choices is different from the number of distinct values actually written to a file.
Do not substitute the domain for observations
The size of an allowed domain and the cardinality observed in a file are separate metrics. Even with Truthound checking a distribution, treating the Parquet statistic as the expected answer can reproduce the same mistake. Keep the footer and the decoded values as separate evidence. [1, 2, 3]
50 choices, only two written values
The public DuckDB reproduction defines an ENUM with 50 members but writes only two of them, while Parquet distinct_count is recorded as 50. This does not mean the values are outside their allowed domain. The value set can be valid while the statistic describes something other than the file's contents. [1]
Compare at the row-group boundary
Record the ENUM domain size, row-group metadata and decoded distinct count separately. Do not use the disputed NDV for cost or quality decisions until they agree. Attach file and row-group identities so that a whole-file count is not accidentally compared with a single row group's statistic. [1, 3]
Count kinds, not repetitions
For an illustrative fixture, define every row as one of two values and require both values to occur. The independently decoded cardinality is 2. Metadata reporting 2 passes this comparison; metadata reporting 50 fails. Unused choices in the ENUM definition do not excuse the failure. If new values actually enter another file, calculate its expected basis again. [1, 3]
Where Truthound fits
Truthound Core's DistinctCountValidator can check the actual distinct count in the decoded value column. Separate tools extract the footer and read the file, and an audit record must align their scopes. Truthound does not repair the writer's Parquet footer. While the statistic is untrusted, decide whether to pay for a direct aggregation or hold the affected use of the file. [1, 2, 3]
Application conditions
Link metadata_ndv and independently decoded decoded_ndv to row_group_id within the same file. Keep the ENUM domain size separate.
Alternatives
Calculate distinct values directly from decoded data instead of relying on the Parquet footer NDV.
Limitations
Truthound does not repair a Parquet writer footer. Separate tools extract metadata and decode the file.
Evidence
- Event checked
- Subject checked
- Modified
