Investigating Dask task fusion when independent chains receive the same key
Preserve the mapping between logical and executable work before relying on output-file existence.
Investigating Dask task fusion when independent chains receive the same key
Verify that optimization retains the lineage from independent logical work to execution results. If chains with different inputs receive the same fused key, later work can replace earlier work. Coincidentally equal results or an existing output file do not establish that every intended task ran. [1, 2, 3]
Scope of the source report
Dask issue 12681 reports collisions when long fused names are shortened with a short hash and inserted without checking existing keys. It describes observations during xarray and icechunk writes and a fusion-disabled alternative. Merely lengthening the name does not prove collisions impossible. [1]
What to inspect in the input
Compare long chains with independent inputs and isolate optimized output when their logical jobs collapse onto the same fused key. Preserve input identity to distinguish deliberate shared computation from an unintended collision. Record both the pre-execution mapping and actual results. [1, 3]
A small acceptance example
Define illustrative chains A and B as consuming different inputs. Distinct execution keys satisfy their independent-ownership condition. A common fused key representing both is counted as 1 collision. Equal output values do not approve merging without separate evidence that the work was intentionally shared. Distinct execution keys for the independent chains give a collision count of 0. [1, 3]
The boundary passed to Truthound
MultiColumnUniqueValidator in Truthound Core 3.3.2 can check the execution-key ownership columns that must actually be unique. Including logical_task_id merely to make every row unique would hide repeated fused keys, so choose the key correctly. Graph repair belongs to Dask; retest a verified fusion-disabled path or an implementation with collision detection. [1, 2, 3]
Application conditions
Retain original logical tasks alongside fused keys, but apply uniqueness to the actual key-ownership columns.
Alternatives
Compare a verified path with fusion disabled, or use an optimizer with collision checks and preserved original keys.
Limitations
A longer hash string does not prove collisions impossible; changes to the execution graph remain Dask's responsibility.
Evidence
- Event checked
- Subject checked
- Modified
