Styxis
Company
Knowledge

Checking empty strings versus NULL across Polars CSV engines

Matching String dtypes do not establish that a quoted empty field retained its meaning.

Value state matters beyond dtype

A String dtype cannot establish preservation of empty and absent states. An empty string may represent an explicitly cleared value, while NULL may represent a value that was not supplied. If the application preserves that distinction, its engine-migration acceptance test must preserve it too. [1, 2, 3]

The quoted empty field

The Polars report writes a single empty string as a quoted CSV field and reads the same file with CPU and GPU execution. The CPU result preserves the empty string; the GPU result contains NULL. Both retain String dtype. A visually blank cell is therefore insufficient evidence of the decoded state. [1]

Connect lexical and decoded states

Record whether the original CSV token is quoted-empty, then compare CPU and GPU outputs as empty, null or text. Separate quoted-empty, unquoted-missing and nonempty source states and join results by record_id. Explicit state columns also make it possible to investigate whitespace-only strings without confusing them with empty strings. [1, 3]

Allowed pairs and a failing transition

Define an illustrative contract in which quoted-empty must decode to empty. The pair (quoted-empty, empty) passes; (quoted-empty, null) fails. The expectation comes from an approved interpretation rule, rather than blindly copying the CPU output. If the application intentionally normalizes empty strings to NULL, declare that different set of allowed pairs. [1, 3]

Checking the state table with Truthound

Truthound Core's ColumnPairInSetValidator can validate allowed combinations of lexical and decoded states. Prepare the state extraction separately and distinguish a NULL value from the ordinary string "null". The check does not repair the GPU parser, but it identifies the particular state transition that violates the contract. [1, 2, 3]

Application conditions

Prepare a record_id, lexical_state and decoded_state table from raw tokens or an independent control, aligned to the same run, file and key scope.

Alternatives

Preserve raw lexical states through a CPU path before applying explicit normalization.

Limitations

  • If the application intentionally normalizes empty strings to NULL, record a separately approved rule. This policy does not impose CPU semantics in every application.

Evidence

Event checked
Subject checked
Modified