Styxis
Company
Knowledge

pandas CSV numeric separators: preserving the meaning of Unicode locale settings

Internationalization settings affect interpretation, not just display. Preserve the raw token and the actual separator used to decode it.

pandas CSV numeric separators: preserving the meaning of Unicode locale settings

Producing a numeric column after changing separator settings does not establish the intended interpretation of an amount. If a parser consumes the wrong character as a separator, a plausible number can remain after the original token's meaning is lost. Inspect the character rule used, rather than accepting conversion success alone. [1, 2, 3]

What the primary source establishes

A pandas report describes Unicode thousands and decimal separators being narrowed to a byte in the C parser, causing a slash or k to be treated as a numeric separator. The author separates execution on a stable version from inspection of main's code and states that the reproducer was not run against main. [1]

The handover decision

Approve the allowed separators for each input locale. Keep numbers interpreted through an unapproved separator out of the historical series until they have been reread from the original. Record raw_token, configured_separator, normalized_token, and parse_status so a plausible numeric result does not conceal the change in meaning. [1, 3]

A reference defined before the output

The illustrative input is 1/234, while the allowed thousands separator is a narrow space. Its expected state is unparsed. Removing the slash and admitting a numeric value changes the input's interpretation. This is a deliberately scoped contract, not a universal statement about slash-separated numbers. [1, 3]

The scoped Truthound check

Truthound Core 3.3.2 can compare approved and observed parse_status values with ColumnPairValidator. A preprocessing layer must preserve the raw text, interpret the separator, and produce those states. If only the converted number remains, this check cannot recover the separator that was wrongly consumed. [1, 2, 3]

Limits and alternative choices

The report does not establish that every pandas version or every Unicode separator is affected. Read the column as text first, then apply an approved locale-specific numeric conversion. If the source contract genuinely allows a slash as a thousands separator, this particular token is not a rejection case. [1, 3]

Application conditions

Retain the original token and independently approved separator policy before preparing conversion-status records.

Alternatives

Read the column as text first, then apply an approved locale-specific numeric conversion.

Limitations

  • The report does not establish that every pandas version or every Unicode separator is affected.

Evidence

Event checked
Subject checked
Modified