Keep Delta partition-filter casts consistent with scan values under ANSI
Apply the same parsing policy to malformed partition strings in full-scan and filtered paths. Preserve raw values and ANSI settings so a cast error stays distinct from a valid empty result.
Interpret one raw value consistently
Partition filtering and returned rows must interpret the same raw partition value consistently. Truthound Core can compare an exported filter outcome with the independently declared scan contract. [1, 2, 3]
NULL scans and ANSI errors
Delta issue #7930 reports master with Spark 4.x returning malformed INT partition string abc as NULL in an unfiltered scan while ANSI-enabled partition filters fail with CAST_INVALID_INPUT. The report distinguishes raw partitionValues casting from the lenient scan path. [1]
Keep execution state and keys
Preserve raw partitionValues, returned state, ANSI setting, and filter result at one snapshot. Compare IS NULL across the selected paths. Do not replace abc with an arbitrary number merely to make the query succeed. [1, 3]
A declared lenient reference
Declare the illustrative scan contract as exposing abc as NULL for key 1. p_int IS NULL therefore expects SUCCESS_KEY_1. That passes; CAST_ERROR fails the consistency condition. It is an explicit lenient contract, not an assertion that malformed input is always acceptable. [1, 3]
Inspect prepared status tags
ColumnPairValidator in Truthound Core Python SDK 3.3.2 can compare prepared non-null status tags. The operator defines the parsing policy and executes each path. The SDK does not alter Spark ANSI settings or Delta partition casts. [1, 2, 3]
Retain invalid source values
Review a supported lenient-parsing path and isolate invalid raw partition records. Include p_int=2 and Boolean controls separately so the response does not generalize a single malformed integer case. [1, 3]
A full scan cannot approve its filter
A complete scan succeeding does not approve its filtered consumer. Keep execution status and selected keys in the same acceptance artifact so a parsing failure cannot be mislabeled a legitimate 0-row result. [1, 3]
Application conditions
Prepare raw partitions, scan state, ANSI setting, and filter outcome at one snapshot.
Alternatives
Review a supported lenient path together with invalid-raw isolation.
Limitations
Replacing malformed source with an arbitrary number prevents checking the original lenient contract.
Evidence
- Event checked
- Subject checked
- Modified
