Styxis
Company
Knowledge

Why ClickHouse NATS checks must separate messages from rows

A successful message receipt does not establish complete landing when one message expands into several rows.

A message is not necessarily a row

Using one unit for message counts and parsed-row counts can hide missing multi-row messages. The audit table supplied to Truthound should distinguish expected and landed rows for each message identity. A ratio mixing broker receipts with table rows cannot explain the failure. [1, 2, 3]

Error columns on a successful message

The public ClickHouse report identifies a NATS stream error-handling path that adds a default to virtual error columns only once when a valid message parses into several rows. The resulting column-length mismatch prevents those rows from reaching the destination. An invalid message and a valid multi-row message are different control cases. [1]

Bind the landing contract to each message

Link expected and received row counts per message, separating valid, invalid and multi-row controls. If an illustrative message contains 2 valid records, a landed count of 2 passes and 0 fails. Recording one successful message receipt does not cancel that failure. [1, 3]

Check before replaying

Truthound Core's ColumnPairValidator can compare approved parsed-row counts with landed counts for the same message. Derive expectations by independently parsing a preserved message, not by recounting the destination. A failed check should not trigger blind replay: existing effects could be duplicated. Establish stable message identities and retry policy first. Truthound does not transmit NATS messages or repair ClickHouse Chunks. [1, 2, 3]

Application conditions

Link expected_rows from independently parsed preserved messages to landed_rows for the same message_id.

Alternatives

Reprocess valid multi-row messages and invalid messages separately, comparing them through stable message keys.

Limitations

  • Truthound neither retransmits NATS messages nor restores ClickHouse Chunks. Check retention and duplicate-handling policy before replay.

Evidence

Event checked
Subject checked
Modified