Bind Hudi expression indexes to functions as well as columns
An expression index must match the query function as well as its source column. Reconcile filters using a different function against an unindexed control at the same snapshot to detect incorrect pruning.
Function identity belongs to the index
An expression index is identified by its function and result type as well as its source column. Truthound Core can check matching keys against an independent function-specific reference after the operator exports comparable results. [1, 2, 3]
A lower-index report
Hudi issue #20255 reports lower(city) index statistics being applied to upper(city) or length(city) filters, causing missing rows or ClassCastException. A follow-up says a linked PR fixes the issue; it does not establish which deployed release contains it. [1]
Record both functions
Record the index definition function and the query filter function. For a different function, compare the selected keys with an index-disabled control over the same snapshot. Preserve errors as errors rather than empty matching sets. [1, 3]
The City_4242 reference
Set city-row.city=City_4242 in an illustrative fixture. upper(city)=CITY_4242 must include city-row. That key passes; empty fails if a lower index incorrectly excludes it. A dataset already entirely lowercase is not evidence that the two functions are equivalent. [1, 3]
Expected keys are supplied externally
DistinctValuesEqualSetValidator in Truthound Core Python SDK 3.3.2 compares the prepared result key column with expected_values. The operator derives function semantics and the control result. The SDK does not change Hudi optimizer function matching. [1, 2, 3]
Approve each function path
Confirm a path that does not apply the mismatched expression index, and maintain controls for each function. If adopting the linked correction, test the real query shape and installed version rather than relying on issue closure. [1, 3]
A column-only inventory is incomplete
The acceptance artifact should preserve function names and result types next to the source field. A source-column-only inventory hides the difference that made this pruning decision unsafe. [1, 3]
Application conditions
Prepare index function/result type and query-function keys at one snapshot.
Alternatives
Verify a path avoiding the mismatched expression index for each function.
Limitations
A shared source column or coincidental result does not prove function equivalence.
Evidence
- Event checked
- Subject checked
- Modified
