Dask left-merge row loss: inspect shuffle buckets at the 128 boundary
Partition scaling tests need both bucket representation boundaries and preservation of left-hand rows.
Dask left-merge row loss: inspect shuffle buckets at the 128 boundary
When a left merge works at a small scale but loses rows with more partitions, increasing input volume alone is a poor diagnostic. Follow rows at the routing-number boundary directly. Connecting a row's assigned bucket with its actual destination creates a starting point for distinguishing numerical representation from other delivery problems. [1, 2, 3]
Scope of the source report
A public Dask report describes left-merge row loss on Windows with a large partition count and proposes bucket integer width and delivery behavior as possible causes beyond 128 partitions. The proposal is distinct from a confirmed fix or independent reproduction of the complete environment. A similar symptom elsewhere does not establish the same cause. [1]
What to inspect in the input
Pin left-hand rows assigned to boundary buckets and compare a smaller setup with one that crosses the boundary. Record each row's raw bucket number, partition count, actual destination, and presence in the final output. If rows still disappear with a valid representation, continue investigating differences in delivery and joining. [1, 3]
A small acceptance example
With 129 partitions in this illustrative setup, valid buckets run from 0 through 128. A bucket of 128 becoming -128 in a narrow signed representation falls outside that range. This small boundary case does not replace the report's full reproduction, and a valid number alone does not prove delivery to the correct worker. [1, 3]
The boundary passed to Truthound
Truthound Core's BetweenValidator can check the prepared bucket column against an operator-defined range of 0 through 128. The component exposes out-of-range numbers; it does not repair shuffling or guarantee row preservation. Derive the range from the actual partition setup and reconcile original left-hand rows with the final output separately. [1, 2, 3]
Application conditions
Derive the allowed bucket range from the actual partition setup and record raw buckets, destinations, and preservation of left-hand rows.
Alternatives
Reconcile raw buckets with actual receiving workers and use a verified integer width and shuffle path.
Limitations
A lower-bound check does not establish correct routing, and the report's proposed cause is distinct from evidence of a confirmed fix.
Evidence
- Event checked
- Subject checked
- Modified
