Checking empty strings versus NULL across Polars CSV engines
Matching String dtypes do not establish that a quoted empty field retained its meaning.
Value state matters beyond dtype
A String dtype cannot establish preservation of empty and absent states. An empty string may represent an explicitly cleared value, while NULL may represent a value that was not supplied. If the application preserves that distinction, its engine-migration acceptance test must preserve it too. [1, 2, 3]
The quoted empty field
The Polars report writes a single empty string as a quoted CSV field and reads the same file with CPU and GPU execution. The CPU result preserves the empty string; the GPU result contains NULL. Both retain String dtype. A visually blank cell is therefore insufficient evidence of the decoded state. [1]
Connect lexical and decoded states
Record whether the original CSV token is quoted-empty, then compare CPU and GPU outputs as empty, null or text. Separate quoted-empty, unquoted-missing and nonempty source states and join results by record_id. Explicit state columns also make it possible to investigate whitespace-only strings without confusing them with empty strings. [1, 3]
Allowed pairs and a failing transition
Define an illustrative contract in which quoted-empty must decode to empty. The pair (quoted-empty, empty) passes; (quoted-empty, null) fails. The expectation comes from an approved interpretation rule, rather than blindly copying the CPU output. If the application intentionally normalizes empty strings to NULL, declare that different set of allowed pairs. [1, 3]
Checking the state table with Truthound
Truthound Core's ColumnPairInSetValidator can validate allowed combinations of lexical and decoded states. Prepare the state extraction separately and distinguish a NULL value from the ordinary string "null". The check does not repair the GPU parser, but it identifies the particular state transition that violates the contract. [1, 2, 3]
Application conditions
Prepare a record_id, lexical_state and decoded_state table from raw tokens or an independent control, aligned to the same run, file and key scope.
Alternatives
Preserve raw lexical states through a CPU path before applying explicit normalization.
Limitations
If the application intentionally normalizes empty strings to NULL, record a separately approved rule. This policy does not impose CPU semantics in every application.
Evidence
- Event checked
- Subject checked
- Modified
