Missing ClickHouse shards: check responding servers, not just DNS
Successful connections and queries can still read the same shard repeatedly and produce an incomplete aggregate.
Ask where a successful response came from
Distributed query success and coverage of the intended shard identities are separate checks. A healthy server response does not prove that the planned shard participated. To connect Truthound, collect execution evidence identifying the responding servers as well as their returned values. [1, 2, 3]
A connection can survive IP reuse
The public ClickHouse report describes Kubernetes restarts reassigning an old IP to another shard. Interaction between DNS caching and the native connection pool can retain a connection to that wrong shard. Servers sharing users and a cluster secret may accept the connection. A normal-looking DNS display does not establish the peer of an already pooled connection. [1]
Link the plan to the participant set
Collect the expected shard set and each response's server identity before and after deployment. Withhold aggregate approval when shards are missing or repeated. Identify the actual instance-to-shard mapping instead of relying only on a hostname string. Record the permitted scope of retries and replica switching too. [1, 3]
Response count is not unique coverage
An illustrative plan requiring 3 distinct shards needs a unique participant count of 3. A unique count of 2 fails even if one server responds repeatedly. A different set with a count of 3 can also be wrong, so the final decision must compare the actual members of the approved shard set. [1, 3]
What follows the set check
Truthound Core's DistinctValuesEqualSetValidator can compare collected participant identities with the approved shard set. Keep repeated-response counts separately from set membership. The check does not clear DNS or connection pools, nor does it replace authentication settings. An operator chooses a connection-refresh procedure after identifying the real peers, then verifies coverage on the same distributed workload again. [1, 2, 3]
Application conditions
Collect the approved shard set and actual responding-server instance-to-shard mapping for each query_run, linking them to the same distributed execution.
Alternatives
Identify the actual server peers, then refresh connection-pool and DNS state through an operational procedure.
Limitations
Operational queries must collect server identities. Truthound does not clear DNS or connection pools and does not replace TLS settings.
Evidence
- Event checked
- Subject checked
- Modified
