Checking Parquet root nodes before using StreamReader
Format-level support for nesting does not imply that every reader API supports nested schemas.
Distinguish the format from the API
Separate Parquet's format capabilities from the schema supported by a particular reader. Choosing a reader from the filename alone can send unsupported structures to a native constructor. Inspect the schema through a safe metadata path before choosing how to read it. [1, 2, 3]
Root fields are not the same as leaves
The Arrow main report describes Parquet StreamReader and StreamWriter constructors failing to check their flat-schema restriction. Iterating root fields using the number of leaves can access beyond the root vector or treat a group as a primitive. The report does not establish the same defect in every Parquet reader. [1]
Route from metadata first
Extract each root field's kind before opening it with the body reader; route nested fields to a supported general Arrow reader. Retain node_kind and leaf_count in the metadata inventory. If a flat downstream table is required, create it under an explicit flattening rule rather than forcing a nested file through a flat-only path. [1, 3]
A primitive-only illustrative contract
Define the StreamReader route to accept only primitive root nodes. A primitive passes; a group is rejected from that route. Truthound Core's InSetValidator can inspect node_kind in safely extracted metadata. Calling the unsafe constructor just to obtain validation input reverses the intended sequence. The extraction tool must also be safe; this validator is not a native-memory security boundary. [1, 2, 3]
Application conditions
Prepare root_field_name, node_kind and leaf_count through original or independent safe metadata extraction, aligned to the same run, file and key scope.
Alternatives
Choose a general Parquet Arrow reader that supports nested schemas.
Limitations
Truthound does not turn an unsafe native constructor into a security boundary. The metadata extraction tool must also be safe.
Evidence
- Event checked
- Subject checked
- Modified
