Styxis
Company
Knowledge

DataFusion UNION ALL: validate declared schemas and executed batches separately

At export time, check that the schema observed in planning matches the type contract of the batch actually received.

A plan is not the delivered output

A planning-time schema check cannot replace a check of the executed result. If a UNION's common type is resolved later, a consumer that inspected DataFrame.schema can receive a different type. Place the Truthound check where the actual batch arrives. [1, 2, 3]

The analyzer chooses another type

The public DataFusion report describes UNION ALL inputs with different types. The initial schema represents the first input, while executed batches use the type selected by the TypeCoercion analyzer. Examples include timestamp units and physical string types. Matching names and displayed values do not establish matching storage units. [1]

Define the approved schema independently

Compare each actual batch schema with the approved export schema. Where necessary, apply the same explicit target cast to every UNION branch. The target comes from the destination contract, not merely the initial plan string. Preserve declared, executed and approved schemas together to locate the boundary of disagreement. [1, 3]

A different unit is a contract failure

For an illustrative export contract requiring ns, an actual ns unit passes and us fails. Representing the same instant does not waive the unit requirement. If conversion is allowed, specify unit conversion and precision-loss policy, then validate the converted result again. [1, 3]

The type information Truthound can see

Truthound Core's TableSchemaValidator compares the received table's columns and types with the approved schema. Arrow physical types or extra metadata may be lost during conversion to a Polars table, so preserve required details in an audit table before conversion. Truthound does not automatically synchronize planner and analyzer schemas. [1, 2, 3]

Application conditions

Preserve declared_schema, executed_schema, approved_export_schema and the actual output batch for the same export.

Alternatives

Apply the same target cast to every UNION branch and inspect the actual batch again.

Limitations

  • Preserve externally any Arrow physical-type metadata that can be lost when converting to a Polars table.

Evidence

Event checked
Subject checked
Modified