Iceberg binary truncate counts bytes: checking ClickHouse partition prefixes
Bytes resembling UTF-8 characters still require byte-based truncation in a binary partition.
Separate strings from binary values
Do not apply character-length rules to a binary partition contract. A readable display does not change the original binary type or its byte-based prefix unit. A lossless hexadecimal representation is clearer than a displayed string in the comparison table supplied to Truthound. [1, 2, 3]
A prefix measured in code points
The public ClickHouse report describes Iceberg binary and string values sharing an internal string path where truncate uses UTF-8 code points. Filtering can then miss rows in correctly partitioned files from another writer. The report also distinguishes a write path that can create the wrong partition value. [1]
An illustrative single-byte prefix
Let the raw bytes be c3bc and require the leading byte only. The expected hex value is c3. A result of c3 passes, while retaining an entire displayed character as c3bc fails. Decoding to a string and re-encoding before comparison can obscure the exact unit distinction being tested. [1, 3]
Prepare evidence at each boundary
Preserve raw bytes, the byte-length policy and the expected prefix in hex. Include binary values that resemble multi-byte UTF-8. Compare the partition value directly, then compare filtered rows with an unpruned full read. Record values at each boundary so a writer error can be separated from a reader error. [1, 3]
The comparison Truthound can perform
Truthound Core's ColumnPairValidator can compare the hex from independent byte arithmetic with the hex extracted from storage and reading paths. Hex is an inspection representation, not a new business encoding. Handle NULL separately. Truthound does not fix Iceberg partition generation or pruning; an inconsistent path needs a correction or verified alternative followed by another read. [1, 2, 3]
Application conditions
Independently calculate the expected prefix from the same source_hex and truncate_length_bytes using byte arithmetic, then link it to observed_prefix_hex.
Alternatives
Use byte-specific prefix calculation and compare against a path without pruning.
Limitations
Hex is a comparison representation. Truthound does not repair the Iceberg binary-partition writer or pruning logic.
Evidence
- Event checked
- Subject checked
- Modified
