Styxis
Company
Knowledge

Checking value parity for V4 Arrow IPC union subset reads

Bind full-read and subset-read controls to the same file, metadata version, union kind and row identity.

Selective reading must preserve values

A subset-read optimization must preserve the values obtained from a full read, including the file's metadata-version contract. Reading fewer byte ranges reduces I/O; it should not change meaning. A full-read path provides a useful control for that optimization. [1, 2, 3]

The V4 and V5 buffer layouts differ

The Arrow report describes a validity-bitmap slot present in V4 union metadata and absent in V5. Recording subset byte ranges under a default V5 layout can select incorrect buffer offsets for a V4 message and corrupt the chosen union values. The reported scope is the combination of metadata version, union layout and partial-field reading. [1]

Fix file identity and align rows

Keep distinct V4/V5 and sparse/dense union controls. Compare selected values from a full read with subset reads using the same row identities. Record file hashes and requested fields, and compare missingness separately from values. Use stable identities rather than position if row order can change. [1, 3]

An illustrative value that changes from 20 to 0

Define a control whose independently verified full-read value is 20. A subset value of 20 passes; 0 fails. This does not prohibit zero generally: it tests disagreement for the same approved file and row. Retain a separate state comparison so that zero and missingness are not collapsed. [1, 3]

The table Truthound can compare

Truthound Core's ColumnPairValidator can compare prepared full_value and subset_value columns. File reading and row alignment remain external work, and structural validation cannot replace value comparison. If failures cluster on V4, investigate a supported full-read path first. Truthound does not recalculate IPC buffer offsets. [1, 2, 3]

Application conditions

Prepare metadata_version, union_kind, row_id, full_value and subset_value from original data or an independent control, aligned to the same run, file and key scope.

Alternatives

Read the complete message body for V4 files and then select the required columns.

Limitations

  • Successful structural validation does not establish value preservation. Truthound does not recalculate IPC buffer offsets.

Evidence

Event checked
Subject checked
Modified