Styxis
Company
Knowledge

Declare timestamp precision before a DuckDB Parquet glob chooses it

Compare actual timestamp values when changing the first file in a glob. An output dtype alone can make already-rounded timestamps appear acceptable.

Declare the mixed-file precision

A first filename is a weak place to store a timestamp precision policy. Declare the precision for the mixed file set, then use Truthound Core to compare exported subsecond integers with an independent source reference. [1, 2, 3]

A first-file schema report

DuckDB issue #28070 reports that the first file in a mixed-unit Parquet glob changes schema selection and can round nanosecond timestamps to microseconds in 2.0. The follow-up report distinguishes union_by_name and a different truncation behavior in 1.5.5. [1]

Preserve order and units

Preserve file order, physical timestamp units, output dtype, and source values. Add a different-unit file before the original first file and compare both dtype and value preservation. A changed dtype alone does not reveal whether previously read values were already rounded. [1, 3]

A 211-nanosecond difference

The source fraction .123456789 is 123456789 nanoseconds. Preserving 123456789 passes a lossless contract. Rounding to .123457 produces 123457000 nanoseconds, changing the value by 211 nanoseconds and failing that contract. The expected integer comes from the original fraction, not the glob output. [1, 3]

Compare exact integer fractions

ColumnPairValidator in Truthound Core Python SDK 3.3.2 can compare prepared non-null integer-nanosecond columns with ==. Extract units and exact values before conversion. A dtype check cannot recover rounded information, and the SDK does not choose DuckDB file schemas. [1, 2, 3]

Account for schema inspection cost

Compare an explicit target schema with union_by_name and account for the cost of examining all file schemas. Do not assume maximum_sample_files is supported by this read_parquet path. A supported option must be verified for the actual interface and version. [1, 3]

Document accepted rounding

Accept a cheaper read path only if it preserves the required precision across file-order changes. If microsecond rounding is acceptable, document it as a changed contract rather than calling the result lossless. [1, 3]

Application conditions

Preserve file order, physical timestamp units, and original fractions.

Alternatives

Verify preservation and cost with an explicit schema and union_by_name.

Limitations

  • A dtype check cannot recover precision already rounded away.

Evidence

Event checked
Subject checked
Modified