Reconcile the Arrow boundary before attributing identifier corruption to Delta
An error discovered after storage need not originate there. Check the original long identifier after a short inline string.
Discovery point versus failure boundary
Reconcile actual strings at the Arrow interchange boundary before attributing corruption discovered after storage to the Delta writer. Replacing only the storage engine can preserve identifiers already scrambled upstream. Record the first boundary where original-text comparison fails. A table joining immediate post-Arrow values with original IDs lets Truthound help distinguish loss before storage. [1, 2, 3]
The smaller reproducer excludes Delta
Polars #28612 reports scrambled long identifiers after an Arrow roundtrip and Delta write. A smaller follow-up reproduces the issue without Delta and points to Utf8View offsets after short inline strings. Preserve that narrower scope rather than generalizing to the entire storage engine. [1]
Hand off values before storage
Keep original identifiers, row IDs, and strings immediately after the Arrow roundtrip. Stop before storage on a mismatch, testing a short-inline then long-string fixture. Use a separate destination, stable row identity, and entire strings rather than display lengths. [1, 3]
An independent original long identifier
The illustrative identifiers are short and a_string_longer_than_12. The second row’s independent expected value is its original a_string_longer_than_12. That output passes; shorta_string_longer_th fails. Plausible row counts and lengths do not establish the same identifier. [1, 3]
String comparison versus buffer repair
ColumnPairValidator in Truthound Python SDK 3.3.2 compares post_arrow_text extracted by the operator immediately after interchange with original_text for the same row_id. Interchange execution and buffer diagnosis are external. Core does not interpret or repair Utf8View offsets or infer that a fix has shipped. [1, 2, 3]
Reread after changing interchange representations
Test verified string interchange representations and library combinations, reconciling both the Arrow boundary and final storage reread. Existing independent boundary checks reduce the need for another audit. A successful final write cannot override an earlier identifier mismatch. [1, 3]
Application conditions
Externally extract stable row IDs, original strings, and immediate post-Arrow values in a separate test.
Alternatives
Reconcile verified interchange representations and versions at both the Arrow boundary and final storage reread.
Limitations
The string comparison does not interpret or repair Utf8View buffers or establish a released fix.
Evidence
- Event checked
- Subject checked
- Modified
