pandas read_json and large integers: choosing exact text without changing the field's purpose
Decide whether a large integer is an identifier, an exact count, or a timestamp before reducing its meaning merely to make parsing succeed.
pandas read_json and large integers: choosing exact text without changing the field's purpose
When a large integer cannot be read, first decide whether the field is used for arithmetic or preserves an identifier's original representation. A different parser successfully reading the file does not make that decision. The field's purpose determines the acceptance criteria for precision and exchange format. [1, 2, 3]
What the primary source establishes
A report describes read_json raising a Value is too big error for a very large integer, while Python json.load followed by DataFrame construction works. The environment description does not identify an exact patch release. [1]
The handover decision
The field owner should specify the required accuracy. For a field that needs no numeric operations, choose a decimal-string preservation path and compare its digits with the original. Retain raw_decimal, admitted_decimal, and field_purpose together rather than treating successful construction of a table as proof that its values kept their intended meaning. [1, 3]
A reference defined before the output
The illustrative contract requires exact decimal text. The accepted form is 1000000000000000000000000, while 1e24 is rejected under this text contract even if it describes the same mathematical value. A numeric contract could legitimately make a different choice; document that choice before comparing the output. [1, 3]
The scoped Truthound check
Truthound Core 3.3.2 can compare the original and admitted representations prepared as exact decimal strings with ColumnPairValidator. Converting the string to a float and back before checking would defeat this purpose. Parser selection and arbitrary-precision arithmetic require separate verification. [1, 2, 3]
Limits and alternative choices
String equality does not prove arithmetic accuracy, and not every large integer should be converted into a string. Use an arbitrary-precision parsing path, specify the business type, and retain the raw representation separately. If consumers require only approximations and permit scientific notation, exact source-string equality is too restrictive. [1, 3]
Application conditions
Prepare the raw and admitted decimal text from independent stages, retaining the field's declared purpose and record identity.
Alternatives
Use an arbitrary-precision parsing path, specify the business type, and retain the raw representation separately.
Limitations
String equality does not prove arithmetic accuracy, and not every large integer should be converted into a string.
Evidence
- Event checked
- Subject checked
- Modified
