Truthound · Ingestion and parsing · 4/4
Find losses at CSV, JSON and columnar file boundaries.
77 articles
pandas json_normalize: retaining original key identity when limiting depth
Standardizing an output column name must not overwrite the source key's type. Otherwise, an archive can lose a working reprocessing path.
pandas Excel date indexes: checking calendar operations without asking users to re-enter valid dates
Distinguish date meaning from the operations offered by its container. A container mismatch should not automatically become a request for users to re-enter valid dates.
pandas CSV converters: separating thread affinity from reproducible mappings
Reproducing a conversion over time requires the lookup snapshot and execution conditions, not just the input file.
pandas categorical dates: verifying the epoch unit written to JSON
Compressing a date column into a categorical representation must not silently change its external unit. Validate the unit contract alongside the values.
OpenTelemetry Rust JSON: checking exemplar values separately from metric values
A readable metric does not prove that its exemplar wire contract survived. Test the two independently.
jq delpaths: agreeing on deletion targets before mixed-sign paths change an array
Hand over deletion targets resolved against the original snapshot, rather than positions in an array that is changing during deletion.
Investigating Dask string truncation at compute with a first-block dtype control
A sufficiently wide declared dtype does not establish preservation of the final concatenated strings.
Check key ownership before treating Dask partition groupbys as final results
A key spanning partitions can produce separate partial aggregates that are mistaken for a complete record.
Dask left-merge row loss: inspect shuffle buckets at the 128 boundary
Partition scaling tests need both bucket representation boundaries and preservation of left-hand rows.
Checking valid boundary keys when Dask divisions contain NaN
Missing-key join semantics and disappearance of valid keys require separate decisions.
Dask map meta differs from the computed dtype: include unmatched keys
A lookup that can create missing values requires separate checks of declared metadata and the computed result type.
Dask left-merge index provenance: do not invent a right-side ID for an unmatched row
Identify the source role of an output index before treating its display value as a business identifier.
Dask isin with string and integer codes: identical display text can select different entities
Compare mixed-object filtering against the original combination of type and value.
Dask merge_asof followed by drop raises KeyError: preserve intermediate dependencies
A column absent from the final output may still be required until the join has finished.
Checking Arrow offsets when ASCII conversion shortens the data buffer
The array offset and the first value offset describe different boundaries.
Checking for unread data after an empty Parquet page
A page with no values is not necessarily the end of its column chunk.
Validate requested rows before calling Arrow Dataset.head(-1)
Do not import pandas negative-head semantics into a different scanning API.
