Styxis
Company
Knowledge

Reused ClickHouse–Iceberg column names: compare filters by field ID

A familiar name can attach a current predicate to statistics belonging to a different historical field.

Identity must outlive the display name

Schema history needs field identities that are more stable than display names. Deleting a column and giving its name to another column makes name-only lineage ambiguous. Before using Truthound, extract each file's mapping among field ID, historical name and current name. [1, 2, 3]

A full scan succeeds while filtering loses rows

The public ClickHouse report reuses a deleted Iceberg column's name. Rows visible in a full read disappear under WHERE. It describes a current-name filter being attached to a different physical column in historical files. Establishing that a file contains a row is separate from establishing that pruning retains it. [1]

Use schema history as the reference

Preserve per-file field IDs and name history, then compare filtered keys with a full read interpreted through the correct IDs. In an illustrative mapping, the current column's approved field ID is 1. Resolving it to 1 passes; resolving it to the historical same-name field ID 2 fails. The reference is identity established by schema history, not the numerical ordering of IDs. [1, 3]

The mapping table Truthound can inspect

Truthound Core's ColumnPairInSetValidator can check approved pairs of field ID and current name. Combine that mapping check with a comparison of the actual filtered key set. Truthound does not directly interpret manifests or Parquet pushdown. A full-read-then-filter path can be evaluated as an interim alternative, with its file scope and cost recorded. [1, 2, 3]

Application conditions

Link independently obtained per-file schema history of field_id, physical_name and current_name with result keys read from those files.

Alternatives

Read complete files by field ID without filter pushdown, then compare matching keys.

Limitations

  • Truthound does not directly interpret Iceberg manifests or Parquet pushdown. Extract per-file mappings with Iceberg tooling first.

Evidence

Event checked
Subject checked
Modified