pandas CSV headers: preserving column identity across UTF-8 BOM handling
Keep the original bytes alongside the normalized header. That makes a later reader's naming difference distinguishable from a real source-schema revision.
pandas CSV headers: preserving column identity across UTF-8 BOM handling
If only the first column of an archived CSV fails to match a new schema, inspect invisible characters before renaming the file's fields. Headers can look identical while input layers differ in whether they retain a BOM. Linking original bytes to approved names identifies the boundary responsible for the mismatch. [1, 2, 3]
What the primary source establishes
A pandas issue reports that read_csv strips the initial UTF-8 BOM, while reading the same bytes through TextIOWrapper leaves it in place. The evidence concerns how input layers interpret a header; it does not establish that the underlying values were corrupted. [1]
The handover decision
The ingestion owner should hand over the raw header, its code points, and the approved canonical name. When an archived file fails to match only its first column, compare that record with the naming history before renaming the dataset. [1, 3]
A reference defined before the output
In this illustrative header contract, remove only the initial BOM and preserve Name. BOM+Name is an explanatory notation for the control character, not a literal data value. Name satisfies the contract; a header still carrying the initial BOM does not. Retaining the original bytes makes that distinction inspectable even after the ingestion library changes. [1, 3]
The scoped Truthound check
Truthound Core 3.3.2 provides ColumnPairValidator for the prepared table of approved and normalized headers, rather than for decoding the raw file. Keep source_encoding and the original code points while comparing the strings to locate the naming boundary. Because null pairs can be skipped, check header presence separately. [1, 2, 3]
Limits and alternative choices
There is no basis here for deleting every U+FEFF throughout the file or claiming that header normalization repairs value semantics. Decode explicitly with UTF-8-SIG when that is the approved policy, and preserve the original separately. If both input paths already share the same normalization policy, an additional discrepancy gate may contribute little. [1, 3]
Application conditions
Prepare the raw header, canonical header, and source encoding independently, retaining the file identity.
Alternatives
Decode explicitly with UTF-8-SIG when that is the approved policy, and preserve the original separately.
Limitations
There is no basis here for deleting every U+FEFF throughout the file or claiming that header normalization repairs value semantics.
Evidence
- Event checked
- Subject checked
- Modified
