Dask array conversion after Parquet projection: align partition and length-vector units
Projection can combine partitions in the execution graph, so align declared partition metadata with the length vector’s actual ownership unit.
Execution units change after projection
Projection can combine partitions in the execution graph, so align declared partition metadata with the length vector’s actual ownership unit. Truthound counts must refer to the same optimization boundary. Selecting fewer columns does not justify promising another team that the original two partitions remain. [1, 2, 3]
The follow-up explanation
Dask #12411 reports a partition-count mismatch in to_dask_array(lengths=True) after selecting a Parquet column subset. A follow-up proposes partition combining by the Parquet optimizer as the explanation. That does not establish the same graph for every dataset or configuration, or a released correction. [1]
Collect at one optimization boundary
Record selected columns, partition IDs before and after optimization, and the actual length vector. Hold array handoff on a unit-count mismatch and test explicit repartitioning. Extract graph and metadata at the same boundary; total row counts remain a separate measurement. [1, 3]
One length owner P0
The illustrative graph explicitly combines two original partitions into one physical partition P0. Expected executed-unit count is 1. Metadata 1 meets that count condition; retained metadata 2 fails. Changing the count alone does not establish row identity or correct partition boundaries. [1, 3]
Actual lengths beyond counts
Truthound Core Python SDK 3.3.2 can compare prepared executed_partition_count and declared_partition_count using ColumnPairValidator with relationship="==". The operator measures both against the same graph. The SDK does not modify the Parquet optimizer or Dask graph, and matching counts still need actual-length and source-ID checks. [1, 2, 3]
Return to source rows before handoff
Compare explicit repartitioning or a verified conversion path, then recheck lengths and source rows. Reuse valid evidence for the same graph and consumer path. A metadata-number patch alone does not establish the actual execution ownership. [1, 3]
Application conditions
Extract actual graph partitions and metadata at the same optimization boundary, summing rows separately.
Alternatives
Compare verified repartitioning or conversion paths and recheck actual lengths and source IDs.
Limitations
Partition-count equality does not prove row identity or content; Core does not modify the optimizer or graph.
Evidence
- Event checked
- Subject checked
- Modified
