Styxis
Company
Knowledge

Debezium PostgreSQL TOAST arrays: do not overwrite an unavailable field with an empty list

Separate an unavailable TOAST array from an actual change to an empty array, then validate the sink’s overwrite decision.

Do not invent a meaning for an absent value

If an absent array field makes a sink replace a customer’s existing choices with an empty list, a collection problem becomes a data change. An unchanged PostgreSQL TOAST value unavailable in the current event is different from source NULL or a real empty array. A Truthound landing check can first examine how the sink interpreted that state, before comparing array contents. [1, 2, 3]

An array schema meets a string placeholder

Debezium #2418 reports a path where a string placeholder for an unavailable TOAST array conflicts with its ARRAY schema. In the reported warn path the field is omitted; in the fail path processing stops. Later header and envelope proposals must be distinguished from a shipped solution. This article analyzes the public report and does not claim reproduction in an installed environment. [1]

Preserve state in the delivery contract

Retain the source key, field presence, value-availability state and chosen sink write action in one delivery record. Stop normalization that automatically replaces an absent field with []. Limit preservation of the previous value to cases supported by an unchanged-state signal; retain actual deletions and NULL updates as separate valid paths. [1, 3]

An overwrite that should fail

For an illustrative fixture, let the previous array be [8,9] and update another column. Define the array as unchanged and unavailable in this event, so expected_write_flag=0. An observed_write_flag of 0 satisfies the overwrite condition. Recording 1 and writing [] creates an unrequested change and fails. If the source actually changed to [], writing it is correct, so the hold rule in this example does not apply. [1, 3]

Check the write decision with Truthound

Truthound Core Python SDK 3.3.2 can compare prepared expected_write_flag and observed_write_flag using ColumnPairValidator with relationship="==". Prepare both flags as non-missing 0 or 1 values and isolate unknown states first. This SDK check does not imply a custom transformation capability in Local MCP, retrieve TOAST values or undo sink writes. [1, 2, 3]

Determine the repair and recovery scope

On a mismatch, stop overwrites for the affected keys and compare the connector version, error-handling configuration and previous sink state. Test a supported unavailable-value representation together with the sink contract instead of adopting a discussion proposal as an established fix. Verify values already overwritten with empty arrays through an authorized source reread; do not reconstruct them by guesswork. [1, 3]

Keep unknown values unresolved

The contract prevents unavailable input from being treated as a user edit. Keep source NULL, empty arrays, unchanged values and unknown states distinct. Without evidence identifying the availability state, do not declare preserving the old value permanently correct; retain a reread task. Recovery is established by reconciling the affected values and write history, not merely by the disappearance of an error log. [1, 3]

Application conditions

Verify unchanged and unavailable states from source logs and connector observations, representing them with non-missing 0/1 audit flags.

Alternatives

Check the supported version and reread policy, or assess fail-mode interruption and reprocessing costs.

Limitations

  • Truthound does not recover TOAST values or change placeholder formats; verify implementation and release of proposed headers separately.

Evidence

Event checked
Subject checked
Modified