Styxis
Company
Knowledge

Avoid counting a Delta AddFile re-add as storage growth

For metadata-only re-adds, reconcile CRC values against active state that preserves file identity rather than counting actions.

Separate action counts from active file counts

A metadata-only re-add does not create a file, so CRC values should be compared with reconciled active file state rather than the number of commit actions. Counting a repeated path as new storage can change capacity alerts and cost allocation without any data growth. The Truthound Core Python SDK can compare two size columns calculated outside the validator for this purpose. [1, 2, 3]

A report about changing only file metadata

Delta issue #8026 reports that Kernel's post-commit SIMPLE CRC construction counts an AddFile as new even when it is a metadata-only re-add of an existing file. Its example starts with one a.parquet of 100 bytes. A Manual Update changes metadata while preserving file identity, yet the reported calculation produces numFiles=2 and tableSizeBytes=200. This is a report and a proposed correction; this article does not claim to have run that path or verified a completed fix. [1]

Fix the reference before reporting capacity

Reconstruct active file state for a re-add commit that preserves file identity, then reconcile both numFiles and tableSizeBytes before using the CRC in capacity reporting. Record the Delta version, path, size, meaning of removal or re-addition, and CRC calculation mode. Reconcile identities that remain active at that version instead of simply summing an action ledger. [1, 3]

The file state establishes 100 bytes

The source example fixes one active a.parquet at 100 bytes, then re-adds the same file with changed metadata. No new file is created, so the approved size is 100 bytes. A CRC of 100 bytes agrees with an active inventory containing that same 100-byte file. A CRC of 200 bytes counts the same file twice and must not be interpreted as actual storage growth or new data arrival. This comparison does not attempt to make a file count of 1 equal an action count of 2. [1, 3]

The boundary of a two-column CRC check

ColumnPairValidator in Truthound Core Python SDK 3.3.2 can compare the non-null values of operator-prepared crc_bytes and reconciled_active_bytes with ==. Reject missing rows through a separate check and compare bytes from the same Delta version. The SDK does not replay the Delta action log or select SIMPLE versus FULL. Equal sizes do not establish file-content checksums or the identity of every file. [1, 2, 3]

Distinguish the proposed fallback from verified behavior

The reporter proposes reusing the existing operation allowlist at the start of post-commit CRC construction and selecting FULL replay for unsupported operations. That is the proposal recorded in the source. Check the supported operations and checksum path in the actual deployed version instead of assuming it has already been applied. The FULL name alone does not make a reference independent, so retain the source log and calculation boundary. [1]

Pause the growth explanation when reconciliation fails

When values differ, distinguish new file creation, a re-add preserving identity, and file replacement. After checking supported operation conditions, active state obtained through FULL replay may serve as a separate reference. The two-column comparison here is a contract that exposes a discrepancy, not a storage-changing repair procedure. Resume capacity alerts or cost reporting when the relationship between active state and CRC can be explained. [1, 3]

Application conditions

Reconcile reconstructed active paths and sizes with the CRC at one Delta version.

Alternatives

Use active state from a supported FULL replay as a separate reference.

Limitations

  • Equal sizes do not establish content checksums or every file identity.

Evidence

Event checked
Subject checked
Modified