Styxis
Company
Knowledge

TIMESTAMPs in BigQuery rejected rows: recovery needs a readable failure ledger

Normal writes and rejected-row handoff are different paths.

The error path also hands off data

Normal writes and rejected-row handoff are different paths. Verify that the timestamp representation satisfies the actual failed-row consumer. A Truthound check cannot account for rows that fail before becoming a readable failure ledger, so include the error path itself in acceptance testing. [1, 2, 3]

Why a producer-only change falls short

Beam #40110 reports a Storage Write API rejected-row TIMESTAMP printed with both a space and T, then failing conversion of the failed-rows output. Several versions were run, while 2.72.0 and 2.73.0 were checked only in source. The report says changing only the producer format is insufficient because consumer branches accept different representations. [1]

A deliberately rejected control

Send an intentionally rejected synthetic row and compare its source timestamp, failed-row representation and consumer reconstruction. Hold completion if the rejection ledger cannot be read. Distinguish SqlTypes.TIMESTAMP from TypeName.DATETIME while retaining source key, rejection reason and original instant. Normal write success cannot validate the failed-rows path. [1, 3]

The same-instant reconstruction condition

The illustrative rejected row R has timestamp 2026-09-03T18:51:43.417Z. Set the flag to 1 only when the actual consumer reconstructs that same instant. A zone-free string containing a space plus T that throws gives 0 and fails. Parsing without an exception still needs an instant-equality check. [1, 3]

A readable failure ledger

Truthound Core Python SDK 3.3.2 can compare prepared required_failed_row_roundtrip_ok and failed_row_roundtrip_ok using ColumnPairValidator with relationship="==". The operator supplies rejection and reconstruction flags, retaining unobserved rejected rows separately instead of treating them as successful absence. The SDK does not repair BigQuery rejection causes or Java formatters. [1, 2, 3]

Reapprove each branch

Test a version addressing the producer and both consumer branches, reconciling normal and rejected writes. A schema without TIMESTAMP or a different consumer has different conditions. Do not present one replacement string as a universal remedy across these paths. [1, 3]

Application conditions

Identify rejected rows and distinguish the SqlTypes.TIMESTAMP and TypeName.DATETIME consumer branches.

Alternatives

Test a correction covering the producer and both consumers, including normal and failed-row paths.

Limitations

  • The SDK does not repair BigQuery rejection or Java formatting; success in one consumer does not establish compatibility with the other.

Evidence

Event checked
Subject checked
Modified