Why ClickHouse NATS checks must separate messages from rows
A successful message receipt does not establish complete landing when one message expands into several rows.
A message is not necessarily a row
Using one unit for message counts and parsed-row counts can hide missing multi-row messages. The audit table supplied to Truthound should distinguish expected and landed rows for each message identity. A ratio mixing broker receipts with table rows cannot explain the failure. [1, 2, 3]
Error columns on a successful message
The public ClickHouse report identifies a NATS stream error-handling path that adds a default to virtual error columns only once when a valid message parses into several rows. The resulting column-length mismatch prevents those rows from reaching the destination. An invalid message and a valid multi-row message are different control cases. [1]
Bind the landing contract to each message
Link expected and received row counts per message, separating valid, invalid and multi-row controls. If an illustrative message contains 2 valid records, a landed count of 2 passes and 0 fails. Recording one successful message receipt does not cancel that failure. [1, 3]
Check before replaying
Truthound Core's ColumnPairValidator can compare approved parsed-row counts with landed counts for the same message. Derive expectations by independently parsing a preserved message, not by recounting the destination. A failed check should not trigger blind replay: existing effects could be duplicated. Establish stable message identities and retry policy first. Truthound does not transmit NATS messages or repair ClickHouse Chunks. [1, 2, 3]
Application conditions
Link expected_rows from independently parsed preserved messages to landed_rows for the same message_id.
Alternatives
Reprocess valid multi-row messages and invalid messages separately, comparing them through stable message keys.
Limitations
Truthound neither retransmits NATS messages nor restores ClickHouse Chunks. Check retention and duplicate-handling policy before replay.
Evidence
- Event checked
- Subject checked
- Modified
