Iceberg–Spark exchange reuse: equal hashes do not establish equal scans
A plausible aggregate still needs evidence that the requested partition actually contributed to it.
Hash equality is not semantic equality
A short hash match cannot replace semantic identity for caching or reuse. Checking only the final sum with Truthound misses cases where different partitions happen to have equal totals. Preserve the requested scope and actual output provenance together. [1, 2, 3]
Different scans reuse one exchange
The public Iceberg reproduction describes SparkBatch equality relying on scan hashes. A collision between different filters lets Spark reuse one shuffle for both scans. The example uses Aa and BB partitions. The issue concerns identity in the reuse decision, not physical files becoming identical. [1]
An illustrative provenance comparison
If an illustrative request selects BB, its approved output provenance is BB. Observed provenance BB passes and Aa fails. That failure remains even when both partition sums coincide. Derive provenance from actually read rows or independent tracing, not by attaching the request label to any returned result. [1, 3]
Supply approved pairs to Truthound
Use different-filter controls with the same hash and link requested partitions to actual result provenance. Truthound Core's ColumnPairInSetValidator can inspect approved requested–observed partition pairs. It does not replace Spark's equality implementation. Compare both values and provenance in a control without exchange reuse, then reassess the corrected path. [1, 2, 3]
Application conditions
Link requested_partition, observed_partition and aggregate_value for the same request, obtaining observed provenance from actual rows or independent tracing.
Alternatives
Inspect partition results through a control without exchange reuse.
Limitations
Truthound does not replace Spark exchange equality. Equal totals can occur by chance, so value comparison alone is insufficient.
Evidence
- Event checked
- Subject checked
- Modified
