Checking Polars Decimal shift fill values when 1.25 becomes 0.01
Decimal accuracy in existing rows and scale conversion in newly filled rows are separate contracts.
Identify the newly filled rows
Correct existing Decimal rows do not establish correct units for new fill values. Shift mixes moved values with newly created ones. Checking only the column dtype can miss an interpretation error at the fill boundary. [1, 2, 3]
A fill value read as physical representation
Polars issue 30544 reports a Decimal fill value extracted as an integer and then interpreted as scaled physical data, turning 1.25 into 0.01. The source distinguishes scale 0, temporal dtypes and the streaming path as controls. Use those differences to narrow the investigation to filled positions rather than generalizing to all Decimal operations. [1]
Compare the same smallest unit
Mark positions created by shift and compare the approved fill amount with the observed amount as integers at the same scale. Converting through float can introduce another rounding issue, so preserve declared scale and a safe conversion path. [1, 3]
125 and 1 represent different smallest-unit amounts
At illustrative scale 2, fill amount 1.25 represents 125 smallest units. A new position representing 125 passes; one representing 1, or 0.01, fails. Derive 125 from the fill policy rather than adopting observed 0.01 as a replacement policy. Correct moved rows do not cancel the failure at the new position. [1, 3]
Truthound checks the prepared comparison table
Truthound Core's ColumnPairValidator can compare expected and observed columns in the same smallest unit. Operators first identify filled positions and scale. The check does not repair the Decimal shift kernel. When choosing an alternative fill path, compare the same Decimal values and generated positions to detect any separate precision loss. [1, 2, 3]
Application conditions
Mark is_inserted_fill at each same row_position and retain expected_minor_units and observed_minor_units as integers at the approved scale.
Alternatives
Explicitly prepare the Decimal fill value at the approved scale.
Limitations
Converting to floating point introduces separate rounding error. Truthound does not repair the Decimal shift kernel.
Evidence
- Event checked
- Subject checked
- Modified
