Check key ownership before treating Dask partition groupbys as final results
A key spanning partitions can produce separate partial aggregates that are mistaken for a complete record.
Check key ownership before treating Dask partition groupbys as final results
Using a partition-local groupby as a global result first requires single-partition ownership of each key. Every local calculation can be correct while still omitting rows held elsewhere. This acceptance condition matters when no later stage combines the partial aggregates. [1, 2, 3]
Scope of the source report
Dask issue 12604 reports split keys under a combination of a list column, prior repartitioning, a derived index and string-conversion configuration. It provides controls removing ingredients and comparisons across versions. Do not identify one setting as the sole established cause or declare every partition aggregate incorrect. [1]
What to inspect in the input
Aggregate the set of partition IDs actually containing each key. Hold partial outputs from final delivery when a key spans partitions. Inspect the locations of a key's rows, not simply the number of partition names in a collection. Compare a global groupby or verified shuffle path, then recheck ownership. [1, 3]
A small acceptance example
If illustrative key K appears in one partition, ownership count 1 satisfies the condition. If its rows span two partitions, each local groupby lacks the complete key and fails this exclusive-ownership requirement. A design that correctly combines partial results later can legitimately use a different requirement. [1, 3]
The boundary passed to Truthound
QueryGroupAggregateValidator in Truthound Core 3.3.2 accepts SQL over a prepared key-partition table with group and aggregate columns. Check distinct partition counts against the required range using lineage obtained from real execution. This does not establish the aggregate function's numerical correctness or guarantee the shuffle implementation. [1, 2, 3]
Application conditions
Collect keys and partition IDs from actual partitions and calculate the distinct partition count for each key in the aggregate query.
Alternatives
Use a global groupby or verified shuffle path and inspect actual partition ownership again.
Limitations
Exclusive key ownership does not establish the numerical correctness of the aggregation function.
Evidence
- Event checked
- Subject checked
- Modified
