Styxis
Company
Knowledge

Dask array conversion after Parquet projection: align partition and length-vector units

Projection can combine partitions in the execution graph, so align declared partition metadata with the length vector’s actual ownership unit.

Execution units change after projection

Projection can combine partitions in the execution graph, so align declared partition metadata with the length vector’s actual ownership unit. Truthound counts must refer to the same optimization boundary. Selecting fewer columns does not justify promising another team that the original two partitions remain. [1, 2, 3]

The follow-up explanation

Dask #12411 reports a partition-count mismatch in to_dask_array(lengths=True) after selecting a Parquet column subset. A follow-up proposes partition combining by the Parquet optimizer as the explanation. That does not establish the same graph for every dataset or configuration, or a released correction. [1]

Collect at one optimization boundary

Record selected columns, partition IDs before and after optimization, and the actual length vector. Hold array handoff on a unit-count mismatch and test explicit repartitioning. Extract graph and metadata at the same boundary; total row counts remain a separate measurement. [1, 3]

One length owner P0

The illustrative graph explicitly combines two original partitions into one physical partition P0. Expected executed-unit count is 1. Metadata 1 meets that count condition; retained metadata 2 fails. Changing the count alone does not establish row identity or correct partition boundaries. [1, 3]

Actual lengths beyond counts

Truthound Core Python SDK 3.3.2 can compare prepared executed_partition_count and declared_partition_count using ColumnPairValidator with relationship="==". The operator measures both against the same graph. The SDK does not modify the Parquet optimizer or Dask graph, and matching counts still need actual-length and source-ID checks. [1, 2, 3]

Return to source rows before handoff

Compare explicit repartitioning or a verified conversion path, then recheck lengths and source rows. Reuse valid evidence for the same graph and consumer path. A metadata-number patch alone does not establish the actual execution ownership. [1, 3]

Application conditions

Extract actual graph partitions and metadata at the same optimization boundary, summing rows separately.

Alternatives

Compare verified repartitioning or conversion paths and recheck actual lengths and source IDs.

Limitations

  • Partition-count equality does not prove row identity or content; Core does not modify the optimizer or graph.

Evidence

Event checked
Subject checked
Modified