Checking valid boundary keys when Dask divisions contain NaN
Missing-key join semantics and disappearance of valid keys require separate decisions.
Checking valid boundary keys when Dask divisions contain NaN
Check valid-key preservation independently of the policy for missing keys. A policy that does not join NaN with NaN is not a reason to lose a populated boundary key. Treating both effects as missing-value handling can misidentify what must be recollected. [1, 2, 3]
Scope of the source report
Dask issue 12653 reports NaN divisions in a float index leading join, merge or repartition paths to lose valid in-range rows or matched values. The author distinguishes missing-key policy from valid-key loss. The report should not become advice to join every missing key as if it identified the same entity. [1]
What to inspect in the input
Test NaN rows together with valid boundary keys. Compare presence and matched values for the valid keys with the source before assessing missing-key policy separately. Record endpoint inclusion and actual divisions. Also separate key existence from correct right-side values so partial-column loss remains visible. [1, 3]
A small acceptance example
Define an illustrative request range that includes valid keys 1 through 4. A separate NaN row gives no reason to remove key 4. Its presence passes that key's preservation check; absence fails. If the requested range intentionally excludes 4, absence is not the same counterexample, so establish the independent request contract first. [1, 3]
The boundary passed to Truthound
ReferentialIntegrityValidator in Truthound Core 3.3.2 can compare expected valid keys with the actual result's reference-key set to expose omissions. Reversing the direction would miss an expected key absent from the result. Retest a separate missing-index path and valid-key repartitioning. Truthound does not repair the join or infer missing values. [1, 2, 3]
Application conditions
Use the source inventory of valid keys as the checked records and the actual output-key set as the reference to detect missing rows.
Alternatives
Handle missing-index rows separately and verify that repartitioning preserves valid keys.
Limitations
Whether NaN keys should join is a separate business decision; missing keys must not automatically represent the same entity.
Evidence
- Event checked
- Subject checked
- Modified
