Checking Dask divisions length separately from correct row values
The relation between partition count and boundary count is not itself a verdict about data loss.
Checking Dask divisions length separately from correct row values
Observe row correctness and partition-navigation metadata as separate conditions. Successfully computed data does not approve the boundaries consumed by later repartitioning. Conversely, an inconsistent boundary count should not immediately be reported as missing rows. [1, 2, 3]
Scope of the source report
Dask issue 12565 presents a divisions-length mismatch when set_index receives npartitions after drop_duplicates. The author limits the finding by saying the data computation appears correct and supplies a separately performed repartitioning control. The demonstrated issue is the metadata relation. [1]
What to inspect in the input
Record partition and boundary counts before handoff, holding downstream splitting that depends on inconsistent metadata. Check boundary ordering and coverage separately. For unknown divisions, distinguish the meaning of None from the length contract. [1, 3]
A small acceptance example
An illustrative collection with 2 partitions requires 3 boundary entries, including endpoints. A count of 3 passes the length relation; 2 fails. Correct count can coexist with incorrect boundary values, so compare actual boundary semantics and row values separately. [1, 3]
The boundary passed to Truthound
ExpressionValidator in Truthound Core 3.3.2 can check this relation in a prepared npartitions and division_count table. Confirm that the table was extracted from the actual collection. Retest separate set_index and repartition operations without claiming the product generates metadata or changes the Dask plan. [1, 2, 3]
Application conditions
Collect the actual partition count and divisions-boundary count at the same graph stage before checking their relationship.
Alternatives
Check a path that separates set_index from repartition, reconciling metadata and values independently.
Limitations
Passing this relationship does not establish sorted or accurate boundary values.
Evidence
- Event checked
- Subject checked
- Modified
