Styxis
Company
Knowledge

Validating distinct counts after variable-offset Polars rolling slices

A selected sequence can lose its original sort order even when cached metadata says otherwise.

Order changes after selection

A per-window positional selection can break the original column's global ordering guarantee. Recompute distinctness from the selected values. If a quality report relies on the cached metric, valid raw values can still produce an incorrect indicator. Prepare Truthound's control input separately from the suspect sort metadata. [1, 2, 3]

The set behind five values

The public reproduction emits [1, 2, 1, 3, 3]. Its reported sort keeps that order and n_unique returns 4, although the actual set is {1, 2, 3}. The report contrasts constant and variable offsets and connects the discrepancy to a retained sorted flag. [1]

Constructing an independent control

Retain each window's selected offset and emitted value, and compare the distinct metric with an independent set calculation. Include both constant-offset and variable-offset controls. Reconstruct the check input from materialized values instead of handing the same Series back to another computation. Window boundaries help distinguish incorrect selection from incorrect metric calculation. [1, 3]

Derive the expected count from values

Fix the illustrative sequence as [1, 2, 1, 3, 3]. Its three distinct values establish an expected count of 3; a reported cardinality of 3 passes and 4 fails. Truthound Core's DistinctCountValidator can apply that expectation to an independently reconstructed value column. Reusing the affected sorted flag can make both calculations fail together, so independence of the checking path is a prerequisite. [1, 2, 3]

Application conditions

Prepare window_end, selected_offset and emitted_value with an independent cardinality control from original data or an independent path. Match the observed run, file and key scope.

Alternatives

Reconstruct values in a fresh Series and build a distinct-count control independent of the sort metadata.

Limitations

  • Reusing the same Series with the affected sorted flag does not provide an independent check.

Evidence

Event checked
Subject checked
Modified