After adding a computed column to a DataFusion sort, recheck export order
A sort in the plan does not establish that the final exported rows remain globally ordered.
Check after the last transformation
When order is an export contract, inspect the actual sequence after the final transformation. Even one computed column can change parallel batch boundaries. Apply a Truthound order check before sorting the result again, otherwise the original path's failure can be hidden. [1, 2, 3]
Projection redistributes sorted batches
The public DataFusion report describes select with a computed column redistributing sorted batches across parallel partitions without restoring global order. Plain-column projection and a small single-batch input are successful controls. A small-sample success does not establish ordering for a larger export. [1]
Keep the inversion positions
Inspect order immediately before export using data spanning batch boundaries. Do not sort first and thereby conceal the defect. Preserve output positions and sort keys; if equal keys are allowed, define tie-break columns in the approved contract. [1, 3]
A synthetic inversion example
An illustrative descending output requires 0 adjacent inversions. The same value set with 2 inversions across batch boundaries fails. That expected count follows from the ordering requirement, not from another result of the sort implementation. Preserving the value set and preserving its sequence are separate conditions. [1, 3]
The check and the next execution path
Truthound Core's DecreasingValidator can inspect monotonicity in the final output column. For group-specific ordering, separate group boundaries first. The check does not insert a merge into DataFusion's plan. Evaluate an explicit global sort after the final projection, then verify its cost and tie-break behavior against the same export contract. [1, 2, 3]
Application conditions
Preserve export_sequence and sort keys in the original output order, linked to batch boundaries of the same export. Do not sort again before inspection.
Alternatives
Perform an explicit global sort after the final computed-column projection.
Limitations
Define an additional tie-break key where equal keys require one. Truthound does not insert DataFusion parallel-merge operations.
Evidence
- Event checked
- Subject checked
- Modified
