Dask date-column assignment changes an integer column: check the scope of the change
Adding a date column must preserve the types and values of columns outside the requested change.
Dask date-column assignment changes an integer column: check the scope of the change
A correctly displayed date column is not enough to approve an assignment. An unrelated integer column becoming floating point can change both the stored schema and the assumptions of downstream operations. Validate the new column and the preservation of existing columns as separate acceptance criteria. [1, 2, 3]
Scope of the source report
A public Dask report describes an unrelated integer column changing to float64 after the result of dd.to_datetime is assigned to a new column. The reporter says a comparison using pandas date conversion does not show the same change. This is the reporter's comparison; a displayed schema difference alone does not establish loss of actual values. [1]
What to inspect in the input
Keep the schema and actual values of a small integer table before adding the date column. Convert and assign the dates on the same input, then compare the existing columns in the computed result. Check whether the conversion result's index matches the original. If an outer alignment introduces missing values, review the reason for type promotion together with the requested operation. [1, 3]
A small acceptance example
In this illustrative request, only the date column b is added and the approved type of a is int64. Retaining int64 satisfies the schema condition; changing a to float64 requires a separate approval. This is a small contract check for an untouched column, not an experiment proving value loss. [1, 3]
The boundary passed to Truthound
Supply the actual computed output table and the approved schema to Truthound Core's TableSchemaValidator, rather than an audit table of before-and-after type names. The component can check preservation of a's type, but it does not repair the assignment graph or explain alignment behavior. The operator must define expected types and reconcile actual values separately. [1, 2, 3]
Application conditions
Prepare the actual computed output dataframe and approved expected_schema. A before-and-after dtype string table is an audit record, not a replacement for the checked data.
Alternatives
Check alignment of the date-conversion result's index and test an assignment path with an explicit existing schema.
Limitations
A displayed schema difference does not by itself establish value loss; inspect the computed result separately.
Evidence
- Event checked
- Subject checked
- Modified
