Styxis
Company
Knowledge

Checking complete dictionary types before Arrow coalesce

Value types alone omit index widths and ordered flags that can determine execution compatibility.

Binding is not execution compatibility

Compatible dictionary value types do not establish a complete execution-type contract. An expression can bind successfully without satisfying the kernel's requirement for identical dictionary types. Decide which representations to retain or decode before execution, rather than repeatedly retrying an unchanged input. [1, 2, 3]

The same int32 values with different indices

The public Arrow report uses int32 dictionary values with int8 and int16 index types. Binding succeeds, but execution reports a type mismatch. The expected path decodes or promotes compatible mixed arguments while retaining dictionary execution for identical types. An inventory containing only value_type cannot express that distinction. [1]

Record the normalization decision

Record index type, value type and ordered flag. Where necessary, use an approved decoding or promotion path and validate the resulting values. Keep original dictionary signatures beside the normalized flat type. Decoding can change memory use and execution characteristics, so check the service's performance constraints before applying it indiscriminately. [1, 3]

A schema contract and a separate value control

Define an illustrative path that normalizes inputs to Int32 before coalesce. Int32 passes this schema condition; a remaining DictionaryMixed representation is held. Truthound Core's TableSchemaValidator can check the approved schema of the actual flat table. Schema acceptance does not prove correct row-wise selection. Retain a separate value and NULL-preservation control, such as the report's [10, 20, 30, null, null] result. [1, 2, 3]

Dispatch remains outside the validator

On failure, correct the normalization input or switch to a supported decode path. Truthound does not insert casts into Arrow's expression binding. Preserve original index types and ordered flags so that performance and ordering assumptions can be revisited. [1, 3]

Application conditions

Prepare input_column, index_type, value_type and ordered_flag together with coalesced output from original data or an independent control. Align the same run, file and key scope.

Alternatives

Normalize to compatible dictionary types or decode into a common value type.

Limitations

  • Truthound does not insert casts into Arrow kernel dispatch. Keeping dictionary encoding is a separate performance and compatibility decision.

Evidence

Event checked
Subject checked
Modified