Styxis
Company
Knowledge

After adding a computed column to a DataFusion sort, recheck export order

A sort in the plan does not establish that the final exported rows remain globally ordered.

Check after the last transformation

When order is an export contract, inspect the actual sequence after the final transformation. Even one computed column can change parallel batch boundaries. Apply a Truthound order check before sorting the result again, otherwise the original path's failure can be hidden. [1, 2, 3]

Projection redistributes sorted batches

The public DataFusion report describes select with a computed column redistributing sorted batches across parallel partitions without restoring global order. Plain-column projection and a small single-batch input are successful controls. A small-sample success does not establish ordering for a larger export. [1]

Keep the inversion positions

Inspect order immediately before export using data spanning batch boundaries. Do not sort first and thereby conceal the defect. Preserve output positions and sort keys; if equal keys are allowed, define tie-break columns in the approved contract. [1, 3]

A synthetic inversion example

An illustrative descending output requires 0 adjacent inversions. The same value set with 2 inversions across batch boundaries fails. That expected count follows from the ordering requirement, not from another result of the sort implementation. Preserving the value set and preserving its sequence are separate conditions. [1, 3]

The check and the next execution path

Truthound Core's DecreasingValidator can inspect monotonicity in the final output column. For group-specific ordering, separate group boundaries first. The check does not insert a merge into DataFusion's plan. Evaluate an explicit global sort after the final projection, then verify its cost and tie-break behavior against the same export contract. [1, 2, 3]

Application conditions

Preserve export_sequence and sort keys in the original output order, linked to batch boundaries of the same export. Do not sort again before inspection.

Alternatives

Perform an explicit global sort after the final computed-column projection.

Limitations

  • Define an additional tie-break key where equal keys require one. Truthound does not insert DataFusion parallel-merge operations.

Evidence

Event checked
Subject checked
Modified