Checking list-element support before converting Arrow data to pandas
A successful top-level conversion does not establish support at a nested list position.
Nesting changes the support boundary
Describe conversion support using both the base type and its nested position. Recording only the name string_view makes a top-level column and a list element appear to use the same path. At a transport boundary, manage type and nested path together. [1, 2, 3]
Types rejected specifically inside lists
The Arrow main report demonstrates pandas conversion failures for list values of string_view, binary_view, decimal32, decimal64 and fixed_size_binary. It describes successful top-level controls and list controls using string, binary and decimal128. Read this expanded type coverage together with earlier fixed_size_binary reports when defining the conversion-support boundary. [1]
What an equivalent representation must preserve
Inventory value types by nested path, normalize through an explicitly supported equivalent representation, and compare elements and NULL structure afterward. Retain original and approved transport types, list lengths, and parent-versus-element missingness. A wider Decimal representation does not by itself prove scale or value preservation. [1, 3]
The allowlist is a transport-preparation check
Define an illustrative route that normalizes list values to string. The normalized type string passes; a remaining string_view fails this route. Truthound Core's InSetValidator can check an approved type set in a prepared inventory. Element and NULL preservation still need separate comparison, and the validator does not extend Arrow's pandas-conversion support table. [1, 2, 3]
Application conditions
Prepare nested_path, source_value_type and approved_transport_type together with conversion results from original data or an independent control, aligned to the same run, file and key scope.
Alternatives
Normalize nested values to supported wider value types and compare scale and NULL preservation.
Limitations
Widening a decimal type does not automatically preserve scale. Distinguish the earlier fixed_size_binary issue from the newly reported type coverage.
Evidence
- Event checked
- Subject checked
- Modified
