Styxis
Company
Knowledge

Investigating Dask string truncation at compute with a first-block dtype control

A sufficiently wide declared dtype does not establish preservation of the final concatenated strings.

Investigating Dask string truncation at compute with a first-block dtype control

Approve the final strings rather than only a declared dtype or block preview. Blocks can have different widths, and the width chosen during final concatenation can truncate later values. If changing only block order changes the stored text, inspect the concatenation boundary before treating it as a display problem. [1, 2, 3]

Scope of the source report

Dask issue 12684 reports compute allocating mixed-width string output using the first block's dtype, truncating longer strings in later blocks. It includes a sufficiently wide declared dtype and a successful control with the longer block first. The evidence concerns block dtypes and final-value differences. [1]

What to inspect in the input

Compare short-block-first and long-block-first executions, then reconcile each final string's length and complete value with its source. Preserve row IDs so reordering does not change which strings are paired. Equal lengths can still hide different text, so length agreement is not value preservation. [1, 3]

A small acceptance example

Suppose illustrative blocks contain 00ab and abcdefgh. The longer source should remain abcdefgh, and an identical result passes that row's preservation condition. A result of abcd calls for investigation of a path unable to retain the later block's width. If intentional fixed-width truncation is the business contract, separate that policy before applying this counterexample. [1, 3]

The boundary passed to Truthound

ColumnPairValidator in Truthound Core 3.3.2 can compare source_text and computed_text aligned by row ID, with missing inputs checked separately. Even when trying explicit conversion of every block to a sufficiently wide common dtype, reconcile the actual compute result again. The validator does not alter Dask memory allocation. [1, 2, 3]

Application conditions

Preserve block order, dtype, and full original strings, comparing them with final strings at the same positions.

Alternatives

Explicitly convert all blocks to a common sufficient dtype and recheck the final values.

Limitations

  • Equal string lengths do not establish equal values, so length checks alone cannot prove complete preservation.

Evidence

Event checked
Subject checked
Modified