Styxis
Company
Knowledge

Check key ownership before treating Dask partition groupbys as final results

A key spanning partitions can produce separate partial aggregates that are mistaken for a complete record.

Check key ownership before treating Dask partition groupbys as final results

Using a partition-local groupby as a global result first requires single-partition ownership of each key. Every local calculation can be correct while still omitting rows held elsewhere. This acceptance condition matters when no later stage combines the partial aggregates. [1, 2, 3]

Scope of the source report

Dask issue 12604 reports split keys under a combination of a list column, prior repartitioning, a derived index and string-conversion configuration. It provides controls removing ingredients and comparisons across versions. Do not identify one setting as the sole established cause or declare every partition aggregate incorrect. [1]

What to inspect in the input

Aggregate the set of partition IDs actually containing each key. Hold partial outputs from final delivery when a key spans partitions. Inspect the locations of a key's rows, not simply the number of partition names in a collection. Compare a global groupby or verified shuffle path, then recheck ownership. [1, 3]

A small acceptance example

If illustrative key K appears in one partition, ownership count 1 satisfies the condition. If its rows span two partitions, each local groupby lacks the complete key and fails this exclusive-ownership requirement. A design that correctly combines partial results later can legitimately use a different requirement. [1, 3]

The boundary passed to Truthound

QueryGroupAggregateValidator in Truthound Core 3.3.2 accepts SQL over a prepared key-partition table with group and aggregate columns. Check distinct partition counts against the required range using lineage obtained from real execution. This does not establish the aggregate function's numerical correctness or guarantee the shuffle implementation. [1, 2, 3]

Application conditions

Collect keys and partition IDs from actual partitions and calculate the distinct partition count for each key in the aggregate query.

Alternatives

Use a global groupby or verified shuffle path and inspect actual partition ownership again.

Limitations

  • Exclusive key ownership does not establish the numerical correctness of the aggregation function.

Evidence

Event checked
Subject checked
Modified