Before Doris Arrow Stream Load, align actual array types with the destination
A valid Arrow IPC representation is not necessarily a compatible input for the destination column's serde.
Transport validity is not destination compatibility
A structurally valid Arrow array can violate the target column's type contract. A Truthound preflight is one step for checking prepared data against the destination schema. It does not replace the server's responsibility to reject incompatible input safely. [1, 2, 3]
A report of a crash rather than rejection
The public Doris reproduction sends an Int32 array to a Boolean target and a Utf8 array to a Decimal target through Arrow Stream Load. It reports a BE SIGSEGV instead of ordinary rejection. This discussion stays within the reported environment and source-analysis scope; it does not claim every Arrow load behaves this way. [1]
Approve the batch that will actually be sent
Before transmission, map actual Arrow types to the destination schema and explicitly check nullability, decimal scale and Boolean representation. Inspect the data immediately before serialization so a schema document cannot approve a differently typed batch. Preserve original types in the audit record when conversion is required. [1, 3]
An illustrative Boolean type counterexample
For an illustrative Boolean destination contract, define Boolean as the approved transport type. A Boolean array passes that type condition. An Int32 array fails even if its values look like truth values. The rule distinguishes completed conversion from raw transmission; it does not prohibit every explicit integer-to-Boolean conversion. [1, 3]
What remains outside Truthound
Truthound Core's ColumnTypeValidator can inspect whether a prepared table has the approved column type. Preserve and compare original Arrow physical types separately before conversion. Server patches, isolated load controls and failed-batch replay discipline remain necessary. A passing check is not proof of native-server memory safety. [1, 2, 3]
Application conditions
Link the actual outgoing batch to its source_arrow_type and destination_type mapping immediately before serialization, preserving pre-conversion types.
Alternatives
Create verified Arrow arrays for each destination type, or use another input format supported by the server.
Limitations
This preflight does not repair unsafe server casts. Server patches and isolation of failed loads remain necessary.
Evidence
- Event checked
- Subject checked
- Modified
