Styxis
Company
Knowledge

DataFusion random-filter rewrites: similar algebra can change the sample rate

Testing one random draw against a range is different from applying two bounds to separate draws.

Evaluation count is part of the meaning

Probabilistic expressions are not the same experiment when a rewrite changes their evaluation count. To monitor selection rates with Truthound, first define when random values are drawn, how often they are drawn and what statistical variation is allowed. [1, 2, 3]

A random condition becomes two comparisons

The DataFusion report rewrites a floor-based random predicate into lower- and upper-bound comparisons, duplicating its volatile argument. The comparisons no longer see the same random draw, changing the expected selection probability. The issue concerns a changed experiment, not one surprising random outcome. [1]

Use a fixed set of draws as a control

Compare the original filter with a control that materializes each random value once. Define an interval appropriate to the sample size before assessing optimizer changes. Record repeated runs and seed support. Widening the interval after seeing the result would replace diagnosis with a manufactured pass. [1, 3]

An illustrative upper bound

Suppose an illustrative experiment expects a selection fraction of 0.10 and predefines an upper bound of 0.12 for a sufficiently large sample. An observation of 0.10 satisfies that upper-bound condition, while 0.28 fails. A real policy also needs a lower bound and must account for sample size and repeated comparisons. Do not copy these numbers to every workload. [1, 3]

A statistical signal is not a causal proof

Truthound Core's BetweenValidator can compare per-run selection fractions with the approved interval. A violation is an investigation signal, not proof of random independence or an optimizer defect. Examine the materialized-draw control and execution plan together, then retest a path that preserves the intended evaluation count. [1, 2, 3]

Application conditions

Preserve sampled_rows, selected_rows and selection_fraction by run_id, choosing the interval before observation for the sample size and repeated-run design.

Alternatives

Materialize each random value once, then evaluate filters on that same sample.

Limitations

  • The allowed interval depends on sample size and experimental design. Truthound does not prove random independence or the rewrite’s mathematical validity.

Evidence

Event checked
Subject checked
Modified