Styxis
Company
Knowledge

Delta deletion vectors do not make a live file disappear from partition inventory

Reconcile partition statistics with active log entries before cleanup touches deletion-vector files. Keeping live-row counts separate from physical file counts prevents an empty-partition misclassification.

Rows and files have different states

A deleted row and an absent file are different states. Partition inventory must retain active files with deletion vectors. Truthound Core can compare an operator-prepared file count against a log-derived reference before cleanup decisions use the statistic. [1, 2, 3]

A live file omitted from inventory

Trino issue #31397 reports that Delta $partitions skips AddFile entries carrying deletion vectors, showing file_count and total_size as zero despite live rows. The report concerns inventory accounting, not a claim that every row in such a file has been deleted. [1]

Reconcile one Delta version

Collect active AddFile entries and their deletion-vector references separately for each partition. Reconcile $partitions counts and sizes at the same Delta version. Do not approve cleanup or an empty-partition classification from a zero statistic alone; retain the file identities behind the count. [1, 3]

One file with a deleted row

Define an illustrative partition with one active file containing two rows, one of which is deleted by a vector. The expected physical inventory is 1 file. A count of 1 passes that condition; 0 omits a still-active file. The remaining 1 row and the 1 file use different units. [1, 3]

Compare prepared file counts

ColumnPairValidator in Truthound Core Python SDK 3.3.2 can compare prepared non-null observed and expected file-count columns with ==. The operator must reconstruct active log state and supply matching versions; the SDK does not decode deletion vectors or remove storage objects. [1, 2, 3]

Separate logical and physical references

Obtain logical live-row counts through an actual live-row query and physical inventory through the transaction log. File min/max values may remain outer bounds rather than exact extrema of surviving rows, so do not use this reconciliation to certify those extrema. [1, 3]

Keep identities behind the count

An inventory discrepancy is a reason to pause the dependent retention decision. Even matching counts do not establish identical file identities. Keep path-level reconciliation available before concluding that a partition is safe to remove. [1, 3]

Application conditions

Prepare active AddFiles, DV references, and partition counts at the same Delta version.

Alternatives

Reconcile live-row queries and transaction-log file inventory separately.

Limitations

  • File inventory agreement does not verify surviving-row extrema or DV contents.

Evidence

Event checked
Subject checked
Modified