Delta deletion vectors do not make a live file disappear from partition inventory
Reconcile partition statistics with active log entries before cleanup touches deletion-vector files. Keeping live-row counts separate from physical file counts prevents an empty-partition misclassification.
Rows and files have different states
A deleted row and an absent file are different states. Partition inventory must retain active files with deletion vectors. Truthound Core can compare an operator-prepared file count against a log-derived reference before cleanup decisions use the statistic. [1, 2, 3]
A live file omitted from inventory
Trino issue #31397 reports that Delta $partitions skips AddFile entries carrying deletion vectors, showing file_count and total_size as zero despite live rows. The report concerns inventory accounting, not a claim that every row in such a file has been deleted. [1]
Reconcile one Delta version
Collect active AddFile entries and their deletion-vector references separately for each partition. Reconcile $partitions counts and sizes at the same Delta version. Do not approve cleanup or an empty-partition classification from a zero statistic alone; retain the file identities behind the count. [1, 3]
One file with a deleted row
Define an illustrative partition with one active file containing two rows, one of which is deleted by a vector. The expected physical inventory is 1 file. A count of 1 passes that condition; 0 omits a still-active file. The remaining 1 row and the 1 file use different units. [1, 3]
Compare prepared file counts
ColumnPairValidator in Truthound Core Python SDK 3.3.2 can compare prepared non-null observed and expected file-count columns with ==. The operator must reconstruct active log state and supply matching versions; the SDK does not decode deletion vectors or remove storage objects. [1, 2, 3]
Separate logical and physical references
Obtain logical live-row counts through an actual live-row query and physical inventory through the transaction log. File min/max values may remain outer bounds rather than exact extrema of surviving rows, so do not use this reconciliation to certify those extrema. [1, 3]
Keep identities behind the count
An inventory discrepancy is a reason to pause the dependent retention decision. Even matching counts do not establish identical file identities. Keep path-level reconciliation available before concluding that a partition is safe to remove. [1, 3]
Application conditions
Prepare active AddFiles, DV references, and partition counts at the same Delta version.
Alternatives
Reconcile live-row queries and transaction-log file inventory separately.
Limitations
File inventory agreement does not verify surviving-row extrema or DV contents.
Evidence
- Event checked
- Subject checked
- Modified
