Styxis
Company
Knowledge

Polars is_in pushdown: checking filter meaning for nulls and negative zero

Compare inclusion decisions for original rows when checking filter equivalence across execution backends.

Ordinary numbers do not establish equivalence

Set-filter equivalence includes decisions for nulls and signed 0, not just ordinary values. A small numeric sample can agree while negation of missing-value membership or handling of negative 0 selects different rows. [1, 2, 3]

A difference at the PyArrow Dataset boundary

Polars issue 30531 reports is_in pushed into a PyArrow Dataset treating nulls and signed 0 differently from the in-memory path. It shows null rows surviving a negated filter and a positive-0 set failing to select negative 0. The source distinguishes scan_parquet and in-memory frames as unaffected controls. [1]

Link filter results back to original rows

Build a small input distinguishing null, positive 0 and negative 0. Compare each original row's inclusion against the approved in-memory semantics. Preserve null as an explicit source-state category rather than leaving a missing comparison value. Include excluded rows in the comparison table too, or omissions will be invisible. [1, 3]

A fixture that should select negative 0

Define illustrative approved semantics treating positive and negative 0 as equal. For a set containing positive 0, expected keep for the negative-0 row is true. A true keep decision for the negative-zero row follows the selected equality policy. A false decision excludes a row that the policy requires treating as equal to positive zero. Define the null-negation case in a separate decision table rather than copying this true expectation to every missing row. [1, 3]

Truthound compares the approved decision table

Truthound Core's ColumnPairValidator can compare expected_keep and observed_keep aligned by original row ID. Both must be explicit nonmissing Booleans; operators choose the null semantics. The validator does not repair pushdown. Check any alternative scan against the same files and decision table before adopting it. [1, 2, 3]

Application conditions

Record input_state, pushdown_kept and policy_kept as explicit non-null states for the same original rows, separately approving NULL and signed-zero policies.

Alternatives

Materialize the inspected scope, then apply an approved in-memory filter.

Limitations

  • Truthound does not select the intended NULL semantics. Do not claim that other Polars scan paths have the same defect.

Evidence

Event checked
Subject checked
Modified