Styxis
Company
Knowledge

A small experiment before deduplicating NULL structs and NULL children

Define whether an absent parent differs from a present parent with a NULL child. Check overall distinct counts and row-level decisions separately.

Missing values with different structures

Define whether absent struct parents and present parents with NULL children are distinct before deduplicating. Combining them merely because both appear missing can erase structure in the original observation. The equality policy must precede deletion. Supply Truthound with uniqueness flags independently derived from the original structures for row-level reconciliation. [1, 2, 3]

Different uniqueness paths on one input

Polars #30599 reports null structs and {a:null} being combined by is_unique while n_unique and other operations distinguish them. The comparison includes different uniqueness paths on the same Series. This article does not prescribe one universal business policy for nested equality. [1]

Keep the original structure before deletion

Preserve parent validity and child values, then independently derive duplicate decisions under the approved equality policy. Retain records when distinct values are marked duplicate. Extract parent validity before flattening and preserve row IDs and original structure. [1, 3]

A reference with 3 distinct values

The illustrative values are {a:null}, null, and {a:1}. With parent absence distinct from child missingness, each occurs once and the first row’s required uniqueness is true. Actual true passes; false fails. An overall distinct count of 3 cannot replace that row-level decision. [1, 3]

Flag comparison does not implement equality

ColumnPairValidator in Truthound Python SDK 3.3.2 reconciles reference_unique_flag independently derived by the operator with actual_unique_flag for the same row_id. Parent validity is extracted before transformation. Core does not repair Polars nested equality, and a count alone does not establish each flag. [1, 2, 3]

Make parent validity explicit

Compare a verified alternative uniqueness path or construct explicit keys retaining parent validity in a separate column. Recheck the same structure and row identities. Existing regressions under the same equality policy can reduce the benefit of another audit. [1, 3]

Application conditions

Extract parent validity and child values before flattening, retaining approved equality semantics and row_id.

Alternatives

Test a verified uniqueness path or explicit keys with parent validity in a separate column.

Limitations

  • The flag comparison does not repair nested equality and cannot be replaced by an overall distinct count.

Evidence

Event checked
Subject checked
Modified