Choosing a NaN policy before moving Polars comparisons to a GPU
When NaN > 0 selects different rows across engines, matching Boolean schemas are not enough.
Choose a policy for NaN first
An engine change must preserve row-selection semantics, including the chosen NaN ordering and comparison policy. A positive-value filter that routes NaN differently changes the population used by downstream aggregates. A Boolean output type says nothing about equivalence of that selection. [1, 2, 3]
What the report asks to preserve
The reported comparison of a single Float64 NaN with zero returns true on the CPU and false on the GPU. The reporter asks the GPU to match Polars CPU ordering semantics. Presenting this as proof of a universal IEEE comparison rule would misstate the expected behavior. [1]
Record which rows are kept
Separate NaN from ordinary numbers and choose between preserving CPU semantics and quarantining NaN. Record each engine's keep decision for identical inputs. A finite-value-only path can isolate NaN before comparison; a path that permits NaN needs its own approved keep policy. Silently dropping quarantined rows also changes quality-metric denominators. [1, 3]
An illustrative NaN-exclusion check
The synthetic contract here is deliberately narrow: the comparison input must contain no NaN values. A count of 0 passes, while 1 identifies a row to quarantine. Truthound Core's NotNaNValidator can check the original numeric column. Passing does not establish equivalence of all engine comparisons, and it does not select the correct policy for applications that intentionally permit NaN. [1, 2, 3]
Application conditions
Prepare input_is_nan, engine and keep_row decisions from original inputs or an independent control, aligned to the same run, file and key scope.
Alternatives
Quarantine NaN in a separate branch or explicitly implement the approved CPU comparison policy.
Limitations
Removing NaN does not make the engines' comparison implementations equivalent. Do not describe the Polars CPU policy as a universal IEEE rule.
Evidence
- Event checked
- Subject checked
- Modified
