DataFusion UNION ALL: validate declared schemas and executed batches separately
At export time, check that the schema observed in planning matches the type contract of the batch actually received.
A plan is not the delivered output
A planning-time schema check cannot replace a check of the executed result. If a UNION's common type is resolved later, a consumer that inspected DataFrame.schema can receive a different type. Place the Truthound check where the actual batch arrives. [1, 2, 3]
The analyzer chooses another type
The public DataFusion report describes UNION ALL inputs with different types. The initial schema represents the first input, while executed batches use the type selected by the TypeCoercion analyzer. Examples include timestamp units and physical string types. Matching names and displayed values do not establish matching storage units. [1]
Define the approved schema independently
Compare each actual batch schema with the approved export schema. Where necessary, apply the same explicit target cast to every UNION branch. The target comes from the destination contract, not merely the initial plan string. Preserve declared, executed and approved schemas together to locate the boundary of disagreement. [1, 3]
A different unit is a contract failure
For an illustrative export contract requiring ns, an actual ns unit passes and us fails. Representing the same instant does not waive the unit requirement. If conversion is allowed, specify unit conversion and precision-loss policy, then validate the converted result again. [1, 3]
The type information Truthound can see
Truthound Core's TableSchemaValidator compares the received table's columns and types with the approved schema. Arrow physical types or extra metadata may be lost during conversion to a Polars table, so preserve required details in an audit table before conversion. Truthound does not automatically synchronize planner and analyzer schemas. [1, 2, 3]
Application conditions
Preserve declared_schema, executed_schema, approved_export_schema and the actual output batch for the same export.
Alternatives
Apply the same target cast to every UNION branch and inspect the actual batch again.
Limitations
Preserve externally any Arrow physical-type metadata that can be lost when converting to a Polars table.
Evidence
- Event checked
- Subject checked
- Modified
