NumPy Unicode strip: character order must not change the stored code
Define chars as a set of codepoints rather than an ordered sequence, then compare with a source-based cleaning reference.
Retain source text with the removal policy
Define chars as a set of codepoints rather than an ordered sequence, then compare with a source-based cleaning reference. Retain original text and removal policy in Truthound inputs. Mixing byte and character units can change multilingual names or stored codes without an exception. [1, 2, 3]
Execution differs from source inspection
NumPy #33125 reports StringDType strip failing to remove ASCII characters placed after multibyte Unicode characters in chars. The initial 2.3.5 run differs from 2.5.4 source inspection. A later comment confirms the byte-length issue on a development branch; that confirmation does not establish a released correction. [1]
Preserve the meaning of an empty result
Preserve original text, approved removal set and codepoints, deriving the reference through an independent string operation. Do not use order-dependent results for key normalization. Define codepoint versus normalized-character semantics first, and keep empty string distinct from NULL with a collision-free audit encoding. [1, 3]
The reference defined by {é,a}
For illustrative text a and removal set {é,a}, expected result is empty regardless of set order. The comparison table encodes that empty string as EMPTY while retaining raw values to avoid collision with literal EMPTY. Observed EMPTY passes; a fails. Empty string and NULL remain different. [1, 3]
A bounded cleaned-string comparison
Truthound Core Python SDK 3.3.2 can compare prepared independently_cleaned_text and cleaned_text with ColumnPairValidator using relationship="==". The operator supplies the source policy and unambiguous empty representation. The SDK does not repair UTF-8 searching or choose Unicode normalization policy. [1, 2, 3]
Retain multilingual boundary tests
Use a verified Python cleaning path and retain multibyte boundary tests under the same policy. A working fixed-width control does not justify converting every input to a fixed width without reviewing output length and storage costs. Reuse valid evidence for an already reconciled consumer path. [1, 3]
Application conditions
Define codepoint versus normalized-character removal semantics and retain raw text.
Alternatives
Use a verified Python cleaning path and retain Unicode boundary regression tests under the same policy.
Limitations
Core compares prepared cleaned strings, does not repair UTF-8 search and requires empty string to remain separate from NULL.
Evidence
- Event checked
- Subject checked
- Modified
