Styxis
Company
Knowledge

Keep Delta partition-filter casts consistent with scan values under ANSI

Apply the same parsing policy to malformed partition strings in full-scan and filtered paths. Preserve raw values and ANSI settings so a cast error stays distinct from a valid empty result.

Interpret one raw value consistently

Partition filtering and returned rows must interpret the same raw partition value consistently. Truthound Core can compare an exported filter outcome with the independently declared scan contract. [1, 2, 3]

NULL scans and ANSI errors

Delta issue #7930 reports master with Spark 4.x returning malformed INT partition string abc as NULL in an unfiltered scan while ANSI-enabled partition filters fail with CAST_INVALID_INPUT. The report distinguishes raw partitionValues casting from the lenient scan path. [1]

Keep execution state and keys

Preserve raw partitionValues, returned state, ANSI setting, and filter result at one snapshot. Compare IS NULL across the selected paths. Do not replace abc with an arbitrary number merely to make the query succeed. [1, 3]

A declared lenient reference

Declare the illustrative scan contract as exposing abc as NULL for key 1. p_int IS NULL therefore expects SUCCESS_KEY_1. That passes; CAST_ERROR fails the consistency condition. It is an explicit lenient contract, not an assertion that malformed input is always acceptable. [1, 3]

Inspect prepared status tags

ColumnPairValidator in Truthound Core Python SDK 3.3.2 can compare prepared non-null status tags. The operator defines the parsing policy and executes each path. The SDK does not alter Spark ANSI settings or Delta partition casts. [1, 2, 3]

Retain invalid source values

Review a supported lenient-parsing path and isolate invalid raw partition records. Include p_int=2 and Boolean controls separately so the response does not generalize a single malformed integer case. [1, 3]

A full scan cannot approve its filter

A complete scan succeeding does not approve its filtered consumer. Keep execution status and selected keys in the same acceptance artifact so a parsing failure cannot be mislabeled a legitimate 0-row result. [1, 3]

Application conditions

Prepare raw partitions, scan state, ANSI setting, and filter outcome at one snapshot.

Alternatives

Review a supported lenient path together with invalid-raw isolation.

Limitations

  • Replacing malformed source with an arbitrary number prevents checking the original lenient contract.

Evidence

Event checked
Subject checked
Modified