Checking complete dictionary types before Arrow coalesce
Value types alone omit index widths and ordered flags that can determine execution compatibility.
Binding is not execution compatibility
Compatible dictionary value types do not establish a complete execution-type contract. An expression can bind successfully without satisfying the kernel's requirement for identical dictionary types. Decide which representations to retain or decode before execution, rather than repeatedly retrying an unchanged input. [1, 2, 3]
The same int32 values with different indices
The public Arrow report uses int32 dictionary values with int8 and int16 index types. Binding succeeds, but execution reports a type mismatch. The expected path decodes or promotes compatible mixed arguments while retaining dictionary execution for identical types. An inventory containing only value_type cannot express that distinction. [1]
Record the normalization decision
Record index type, value type and ordered flag. Where necessary, use an approved decoding or promotion path and validate the resulting values. Keep original dictionary signatures beside the normalized flat type. Decoding can change memory use and execution characteristics, so check the service's performance constraints before applying it indiscriminately. [1, 3]
A schema contract and a separate value control
Define an illustrative path that normalizes inputs to Int32 before coalesce. Int32 passes this schema condition; a remaining DictionaryMixed representation is held. Truthound Core's TableSchemaValidator can check the approved schema of the actual flat table. Schema acceptance does not prove correct row-wise selection. Retain a separate value and NULL-preservation control, such as the report's [10, 20, 30, null, null] result. [1, 2, 3]
Dispatch remains outside the validator
On failure, correct the normalization input or switch to a supported decode path. Truthound does not insert casts into Arrow's expression binding. Preserve original index types and ordered flags so that performance and ordering assumptions can be revisited. [1, 3]
Application conditions
Prepare input_column, index_type, value_type and ordered_flag together with coalesced output from original data or an independent control. Align the same run, file and key scope.
Alternatives
Normalize to compatible dictionary types or decode into a common value type.
Limitations
Truthound does not insert casts into Arrow kernel dispatch. Keeping dictionary encoding is a separate performance and compatibility decision.
Evidence
- Event checked
- Subject checked
- Modified
