Checking Arrow offsets when ASCII conversion shortens the data buffer
The array offset and the first value offset describe different boundaries.
Checking Arrow offsets when ASCII conversion shortens the data buffer
When transformation shortens a buffer, check the physical boundaries referenced by offsets as well as logical string length. An array offset of 0 does not imply that the first value offset is 0; confusing them can reuse old positions in a new buffer. [1, 2, 3]
Scope of the source report
Arrow issue 54906 reports invalid ASCII case-conversion output for valid input whose array offset is 0 but first value offset is nonzero. It points to retained offsets while transformed data starts at the new buffer's beginning, and provides unaffected UTF-8 controls. This does not establish an issue in every string transformation. [1]
What to inspect in the input
Extract input and output buffer lengths and offset ranges together, and do not convert an output failing native validation into a downstream table. Run native array validation first and export only safely obtainable boundary information. Reading an already invalid object to prepare inspection data can repeat the unsafe operation. [1, 3]
A small acceptance example
An illustrative ABC output has a 3-byte buffer. Final offset 3 is within bounds; reusing offset 8 from an earlier buffer goes beyond the end and fails. Obtain the expected boundary from the actual new buffer length. This comparison alone does not establish monotonicity of every offset or correctness of string content. [1, 3]
The boundary passed to Truthound
ColumnPairValidator in Truthound Core 3.3.2 can compare a safely extracted maximum offset with buffer length using an ordering relationship. It is not an Arrow repair tool and does not replace native validation. Select a verified conversion path that rebases offsets to the new buffer, then recheck both boundaries and content. [1, 2, 3]
Application conditions
Run native validation first and prepare only safely obtained offsets and buffer lengths for the check table.
Alternatives
Use a verified transformation path that normalizes offsets for the new buffer.
Limitations
A tabular bounds check cannot safely repair an invalid Arrow object; native validation must come first.
Evidence
- Event checked
- Subject checked
- Modified
