Styxis
Company
Knowledge

Checking Arrow offsets when ASCII conversion shortens the data buffer

The array offset and the first value offset describe different boundaries.

Checking Arrow offsets when ASCII conversion shortens the data buffer

When transformation shortens a buffer, check the physical boundaries referenced by offsets as well as logical string length. An array offset of 0 does not imply that the first value offset is 0; confusing them can reuse old positions in a new buffer. [1, 2, 3]

Scope of the source report

Arrow issue 54906 reports invalid ASCII case-conversion output for valid input whose array offset is 0 but first value offset is nonzero. It points to retained offsets while transformed data starts at the new buffer's beginning, and provides unaffected UTF-8 controls. This does not establish an issue in every string transformation. [1]

What to inspect in the input

Extract input and output buffer lengths and offset ranges together, and do not convert an output failing native validation into a downstream table. Run native array validation first and export only safely obtainable boundary information. Reading an already invalid object to prepare inspection data can repeat the unsafe operation. [1, 3]

A small acceptance example

An illustrative ABC output has a 3-byte buffer. Final offset 3 is within bounds; reusing offset 8 from an earlier buffer goes beyond the end and fails. Obtain the expected boundary from the actual new buffer length. This comparison alone does not establish monotonicity of every offset or correctness of string content. [1, 3]

The boundary passed to Truthound

ColumnPairValidator in Truthound Core 3.3.2 can compare a safely extracted maximum offset with buffer length using an ordering relationship. It is not an Arrow repair tool and does not replace native validation. Select a verified conversion path that rebases offsets to the new buffer, then recheck both boundaries and content. [1, 2, 3]

Application conditions

Run native validation first and prepare only safely obtained offsets and buffer lengths for the check table.

Alternatives

Use a verified transformation path that normalizes offsets for the new buffer.

Limitations

  • A tabular bounds check cannot safely repair an invalid Arrow object; native validation must come first.

Evidence

Event checked
Subject checked
Modified