Styxis
Company
Knowledge

Validating literal dotted names separately from nested ORC paths

Preserve field identity before comparing a top-level a.b column with a child b inside struct a.

A display path is not a complete identity

Do not merge different fields merely because their display paths use the same dotted string. A literal top-level name a.b and child b inside struct a are separate fields. If the audit table collapses them first, it cannot identify which value changed later. [1, 2, 3]

NULL values from a full scan

The public ORC report stores [1, 2] in literal a.b and [10, 20] in nested a/b. A full ORCFile read preserves both, while a full Dataset scan replaces a.b with NULL. The result passes validate(full=True). The report distinguishes this full-scan problem from redefining precedence for ambiguous string selectors. [1]

Create unambiguous field identifiers

Record literal and nested identities separately, then compare values and missingness against a full ORCFile control. Store the identity kind and path components in separate fields. Align row identities as well, so that the comparison identifies where the two originally present values disappeared. [1, 3]

An illustrative literal-field failure

Define a fixture where both rows of literal a.b are present. 2 non-null values pass; 0 fails. The operational check should also align each field and row with its expected state rather than stopping at the count. Truthound Core's ColumnPairInSetValidator can check prepared field-identity tokens against allowed value states. It should not be described as independently interpreting nested ORC structures. [1, 2, 3]

Consider the cost of the alternate reader

Using the report's full ORCFile control requires considering whole-file materialization cost. After switching paths, verify preservation of both literal and nested identities. A symptom disappearing when field order changes is an observation, not a durable solution to the field-identity contract. [1, 3]

Application conditions

Prepare field_identity_kind, field_path, expected_value and observed_value from original data or an independent control, aligned to the same run, file and key scope.

Alternatives

Read the entire file with ORCFile, then select and filter in memory.

Limitations

  • The source concerns a full scan, not a new precedence definition for every string selector. Passing validate(full=True) did not guarantee content accuracy.

Evidence

Event checked
Subject checked
Modified