pandas categorical dates: verifying the epoch unit written to JSON
Compressing a date column into a categorical representation must not silently change its external unit. Validate the unit contract alongside the values.
pandas categorical dates: verifying the epoch unit written to JSON
Changing the internal container for a date does not necessarily change the unit expected by a JSON consumer. A producer may preserve an instant in a different numeric resolution while the consumer interprets it as another time. Check both stored date values and index labels against the external unit contract. [1, 2, 3]
What the primary source establishes
A report describes categorical dates being written at their internal resolution instead of the requested date_unit on the epoch JSON path. The same report separates an exception for sparse dates and an unaffected ISO path. [1]
The handover decision
Compare date values and index labels in archived JSON separately with a normalized time reference. When the internal dtype changes, recheck the unit assumed by historical replay consumers. Keep raw_datetime, container_dtype, requested_unit, and serialized_epoch in the comparison table. [1, 3]
A reference defined before the output
The illustrative reference instant is 2020-01-01 UTC, whose seconds-based contract value is 1577836800. The normal representation remains 1577836800. Writing 1577836800000000000 into that same seconds field fails the contract; a much larger number cannot be accepted merely because it could represent the same instant in another unit. [1, 3]
The scoped Truthound check
Truthound Core 3.3.2 can compare a seconds-based reference with the serialized value using ColumnPairValidator. Establish the same unit on both sides first and avoid a lossy conversion of the large integers to floating point. The comparison does not determine time zones or generate epochs. [1, 2, 3]
Limits and alternative choices
The check does not automatically determine time-zone or calendar meaning, or fix the sparse-date exception. Serialize an explicitly converted datetime representation, or verify ISO output against the consumer's contract. A large integer is not itself erroneous if the actual exchange contract and consumer both use nanoseconds. [1, 3]
Application conditions
Prepare the reference instant in the approved external unit and preserve the container dtype and requested unit for each output field.
Alternatives
Serialize an explicitly converted datetime representation, or verify ISO output against the consumer's contract.
Limitations
The check does not automatically determine time-zone or calendar meaning, or fix the sparse-date exception.
Evidence
- Event checked
- Subject checked
- Modified
