Styxis
Company
Knowledge

Checking list-element support before converting Arrow data to pandas

A successful top-level conversion does not establish support at a nested list position.

Nesting changes the support boundary

Describe conversion support using both the base type and its nested position. Recording only the name string_view makes a top-level column and a list element appear to use the same path. At a transport boundary, manage type and nested path together. [1, 2, 3]

Types rejected specifically inside lists

The Arrow main report demonstrates pandas conversion failures for list values of string_view, binary_view, decimal32, decimal64 and fixed_size_binary. It describes successful top-level controls and list controls using string, binary and decimal128. Read this expanded type coverage together with earlier fixed_size_binary reports when defining the conversion-support boundary. [1]

What an equivalent representation must preserve

Inventory value types by nested path, normalize through an explicitly supported equivalent representation, and compare elements and NULL structure afterward. Retain original and approved transport types, list lengths, and parent-versus-element missingness. A wider Decimal representation does not by itself prove scale or value preservation. [1, 3]

The allowlist is a transport-preparation check

Define an illustrative route that normalizes list values to string. The normalized type string passes; a remaining string_view fails this route. Truthound Core's InSetValidator can check an approved type set in a prepared inventory. Element and NULL preservation still need separate comparison, and the validator does not extend Arrow's pandas-conversion support table. [1, 2, 3]

Application conditions

Prepare nested_path, source_value_type and approved_transport_type together with conversion results from original data or an independent control, aligned to the same run, file and key scope.

Alternatives

Normalize nested values to supported wider value types and compare scale and NULL preservation.

Limitations

  • Widening a decimal type does not automatically preserve scale. Distinguish the earlier fixed_size_binary issue from the newly reported type coverage.

Evidence

Event checked
Subject checked
Modified