Reused ClickHouse–Iceberg column names: compare filters by field ID
A familiar name can attach a current predicate to statistics belonging to a different historical field.
Identity must outlive the display name
Schema history needs field identities that are more stable than display names. Deleting a column and giving its name to another column makes name-only lineage ambiguous. Before using Truthound, extract each file's mapping among field ID, historical name and current name. [1, 2, 3]
A full scan succeeds while filtering loses rows
The public ClickHouse report reuses a deleted Iceberg column's name. Rows visible in a full read disappear under WHERE. It describes a current-name filter being attached to a different physical column in historical files. Establishing that a file contains a row is separate from establishing that pruning retains it. [1]
Use schema history as the reference
Preserve per-file field IDs and name history, then compare filtered keys with a full read interpreted through the correct IDs. In an illustrative mapping, the current column's approved field ID is 1. Resolving it to 1 passes; resolving it to the historical same-name field ID 2 fails. The reference is identity established by schema history, not the numerical ordering of IDs. [1, 3]
The mapping table Truthound can inspect
Truthound Core's ColumnPairInSetValidator can check approved pairs of field ID and current name. Combine that mapping check with a comparison of the actual filtered key set. Truthound does not directly interpret manifests or Parquet pushdown. A full-read-then-filter path can be evaluated as an interim alternative, with its file scope and cost recorded. [1, 2, 3]
Application conditions
Link independently obtained per-file schema history of field_id, physical_name and current_name with result keys read from those files.
Alternatives
Read complete files by field ID without filter pushdown, then compare matching keys.
Limitations
Truthound does not directly interpret Iceberg manifests or Parquet pushdown. Extract per-file mappings with Iceberg tooling first.
Evidence
- Event checked
- Subject checked
- Modified
