Checking for unread data after an empty Parquet page
A page with no values is not necessarily the end of its column chunk.
An empty segment is not the end of the file
Separate a zero-value page from completion of the whole column chunk. Receiving zero rows does not prove that the business dataset is empty; later pages may remain unread. A fixed control file and an independent reader can test that interpretation. [1, 2, 3]
A three-row control from the same file
The PyArrow 26.0.0 report describes treating a data page with num_values=0 as the end of a chunk. Its reproduction returns zero rows in PyArrow while DuckDB reads values 1, 2 and 3. It separately notes that a file with only some affected columns can fail in another way. [1]
Compare counts and identities separately
Retain a control file with a valid page after an empty page. Compare reader outputs and avoid automatically classifying zero rows as a legitimate empty file. Keep file hashes and row_group_id identities to bind the comparison. After locating the count discrepancy, also compare key sets: equal counts can conceal duplicate and missing rows. [1, 3]
The exact count belongs to the decoded table
Define an illustrative approved file with three independently confirmed rows. Three decoded rows pass this count condition; zero fails. With Truthound Core's RowCountValidator and exact_count=3, validate the actual decoded data table. Applying RowCountValidator to a one-row audit table containing declared_rows and decoded_rows does not check the underlying file's record count. [1, 2, 3]
Decide recovery after the check
Use independent reader evidence alongside footer metadata rather than treating a footer number as infallible. Once a supported path reads the control correctly, determine the scope of reingestion. Truthound does not repair continuation after an empty Parquet page, so retain the validation decision separately from the recovery execution. [1, 3]
Application conditions
Use the row count of a fixed control file verified by an independent reader. Apply exact_count to the actual decoded data table associated with file_id and row_group_id, not to a single-row audit containing only count values.
Alternatives
Read the same file with an independent reader to obtain approved rows and keys.
Limitations
The file footer can itself be inaccurate, so an independent reader control is necessary. Truthound does not repair the Parquet page reader.
Evidence
- Event checked
- Subject checked
- Modified
