Styxis
Company
Knowledge

Dask left-merge index provenance: do not invent a right-side ID for an unmatched row

Identify the source role of an output index before treating its display value as a business identifier.

Dask left-merge index provenance: do not invent a right-side ID for an unmatched row

Do not assume the displayed merge index is a stable business key. Right-side indexes for matched rows and left-side indexes for unmatched rows are different kinds of identity even if the combined values look unique. A downstream join can consequently refer to the wrong entity. [1, 2, 3]

Scope of the source report

Dask issue 12564 reports a left merge using left-side indexes for some unmatched rows when the right frame contains empty partitions, while other rows use right-side indexes. It does not claim that retaining unmatched rows in a left join is itself incorrect. [1]

What to inspect in the input

Preserve the left business key and right row ID as separate columns. Leave the right ID absent for an unmatched row and define any downstream index explicitly. Distinguish display-index and reference-key roles in the handoff. Similar printed values do not authorize substituting their provenance. [1, 3]

A small acceptance example

If illustrative left key C has no matching right row, expected right-ID state is missing. Retaining missing preserves absence of a relationship. Copying left C into the right-ID field invents a match and fails. The word missing is explanatory absence notation, not an instruction to delete the unmatched row. [1, 3]

The boundary passed to Truthound

ConditionalExpressionValidator in Truthound Core 3.3.2 can check prepared right_row_id state conditional on match_status. Normalize inputs without confusing actual null with the word missing. Consider an explicit new index while retaining both source IDs as value columns. Truthound does not execute the join or create its provenance evidence. [1, 2, 3]

Application conditions

Separate left row identity, right-match status, and right identity, reflecting actual missingness in the condition.

Alternatives

Create an explicit new index after joining and retain both sides' identifiers as value columns.

Limitations

  • This is not a recommendation to delete legitimate unmatched rows from a left join.

Evidence

Event checked
Subject checked
Modified