Styxis
Company
Knowledge

Bind Hudi expression indexes to functions as well as columns

An expression index must match the query function as well as its source column. Reconcile filters using a different function against an unindexed control at the same snapshot to detect incorrect pruning.

Function identity belongs to the index

An expression index is identified by its function and result type as well as its source column. Truthound Core can check matching keys against an independent function-specific reference after the operator exports comparable results. [1, 2, 3]

A lower-index report

Hudi issue #20255 reports lower(city) index statistics being applied to upper(city) or length(city) filters, causing missing rows or ClassCastException. A follow-up says a linked PR fixes the issue; it does not establish which deployed release contains it. [1]

Record both functions

Record the index definition function and the query filter function. For a different function, compare the selected keys with an index-disabled control over the same snapshot. Preserve errors as errors rather than empty matching sets. [1, 3]

The City_4242 reference

Set city-row.city=City_4242 in an illustrative fixture. upper(city)=CITY_4242 must include city-row. That key passes; empty fails if a lower index incorrectly excludes it. A dataset already entirely lowercase is not evidence that the two functions are equivalent. [1, 3]

Expected keys are supplied externally

DistinctValuesEqualSetValidator in Truthound Core Python SDK 3.3.2 compares the prepared result key column with expected_values. The operator derives function semantics and the control result. The SDK does not change Hudi optimizer function matching. [1, 2, 3]

Approve each function path

Confirm a path that does not apply the mismatched expression index, and maintain controls for each function. If adopting the linked correction, test the real query shape and installed version rather than relying on issue closure. [1, 3]

A column-only inventory is incomplete

The acceptance artifact should preserve function names and result types next to the source field. A source-column-only inventory hides the difference that made this pruning decision unsafe. [1, 3]

Application conditions

Prepare index function/result type and query-function keys at one snapshot.

Alternatives

Verify a path avoiding the mismatched expression index for each function.

Limitations

  • A shared source column or coincidental result does not prove function equivalence.

Evidence

Event checked
Subject checked
Modified