Dask isin with string and integer codes: identical display text can select different entities
Compare mixed-object filtering against the original combination of type and value.
Dask isin with string and integer codes: identical display text can select different entities
A filter's reference is the original type and value, not only displayed text. When string and numeric codes identify different entities, treating them as equivalent can reverse selection. Do not discard source types because the same digit appears on a screen. [1, 2, 3]
Scope of the source report
Dask issue 12612 reports mixed-object isin selecting an integer row with a string needle differently from pandas. The author suggests possible string coercion but does not establish the internal conversion path. A difference in this example is not a confirmed root cause for every object column. [1]
What to inspect in the input
Build a membership truth table preserving original types and compare selected row IDs for string and integer needles separately. Save type tags and values before transformation to locate whether the distinction was already lost upstream. Record the reference dtype as well. [1, 3]
A small acceptance example
An illustrative contract distinguishes string '1' from integer 1. A string needle must not select the integer row, so false is the expected result. A true selection mixes different entities and fails. If the business explicitly normalizes both codes to one identity, define the expectation under that separate policy. [1, 3]
The boundary passed to Truthound
ColumnPairValidator in Truthound Core 3.3.2 can compare independently expected type-aware membership with actual selected values. Preserve type distinctions while preparing the table; the product does not establish Dask's conversion root cause. After explicit typing or type-tagging changes, compare the same pair of needles again. [1, 2, 3]
Application conditions
Retain raw values together with their type tags and define membership independently of the output.
Alternatives
Declare types during ingestion and retain both type tags and values for codes that must remain distinct.
Limitations
The report does not establish behavior of every object column or confirm the internal coercion mechanism.
Evidence
- Event checked
- Subject checked
- Modified
