Truthound · Ingestion and parsing · 2/4
Find losses at CSV, JSON and columnar file boundaries.
77 articles
A 10-byte varint for negative Go INT32: the exchange boundary hidden by positive tests
Check wire width and readback separately for each sign.
Beam Go INT16 wire changes: inspect the bytes of 999 and existing stored rows
Check both the new cross-SDK wire contract and migration of existing Go bytes.
Large IDs in Debezium asNumber: a Number object does not establish exact identity
Creating a numeric object does not establish preservation of an exact identifier.
Check Trino unsigned Parquet decoding beyond the signed boundary
Reading a Parquet unsigned integer into bigint does not establish value preservation. Reconcile keys against independently decoded annotations to detect sign changes at the signed boundary.
Read final Pinot metadata before accepting an empty Trino split
Exclude a Pinot split from successful aggregation when final metadata reports an error, even if its data blocks are empty. Keep missing metadata distinct from a valid empty result so a low count cannot conceal a missing segment.
Declare Parquet statistic order alongside Trino min and max
Before consuming Parquet min/max, check each leaf's column_orders declaration against schema order. Passing the declaration check still leaves the numerical correctness of the bounds to independent validation.
Review the coordinate-axis contract behind Iceberg default CRS compatibility
Evaluate a default CRS and table CRS through the file format's coordinate order and a reviewed pair, rather than names alone. A documented compatibility convention does not establish interchangeability for every CRS.
Project non-null union branches when validating the Hudi file-group reader
Reconcile file-group projection of an Avro union with the legacy path using branch values and member representations. Success for a top-level name cannot approve an untested nested branch.
Check Hudi narrow-integer write, read, and statistics readiness separately
Check write, read, and statistics readiness separately for narrow-integer bulk_insert. Keep ShortType and ByteType paths distinct; file presence or a successful ordinary insert cannot approve statistics-dependent consumption.
Validate NULL array elements across Hudi list-encoding paths
Arrays containing NULL elements need a round-trip check through each consumer path after the write succeeds. Track parent validity, element positions, and NULL state separately to expose empty-array substitution or lost elements.
Approve remote time-decoding read budgets alongside dtype safety
Evaluate open-time improvements using both request counts and preservation of time ranges. Keep actual radar ranges distinct from general decoder boundaries.
Approve integer missing sentinels and physical-value decoding per variable
A global mask_and_scale setting should not silently decide distinct representation contracts for different variables.
Distinguish whole-column format inference failure under strict=False
One invalid row differs from a column with no inferable format. Keep an explicit format beside actual completion status.
Polars decimal-comma CSV: reconcile each scientific token from the actual file
Reconcile the complete numeric token for every output row so previous-row buffer contents are not approved as valid formatting.
Reconcile the Arrow boundary before attributing identifier corruption to Delta
An error discovered after storage need not originate there. Check the original long identifier after a short inline string.
NumPy Unicode strip: character order must not change the stored code
Define chars as a set of codepoints rather than an ordered sequence, then compare with a source-based cleaning reference.
NumPy fixed-width replace: input width must not redefine the search and replacement
Preserve the original search and replacement arguments so input storage width cannot redefine cleaning policy.
True in a Dask object column: identical text is not an identical Boolean state
Use original value types and missingness semantics as the handoff contract rather than display text.
Passing a uint8 mask through Dask: separate creation from completed deserialization
Check dtype and fill semantics through a real serialization roundtrip.
Keep a target type for all-NULL Parquet struct children in DuckDB
Establish cast completion before checking the returned type and NULL positions of a NULL-only struct child. An invalidated connection is not a successfully typed missing-value result.
