Preserve JSON object boundaries when checking ClickHouse distinct results
Use structural controls to check whether distinct merges JSON documents with different object boundaries. If comparison tokens stand in for documents, define key ordering and NULL semantics before accepting the count.
Structure defines document identity
JSON equality must preserve array-element and object boundaries, not just a sequence of scalar values. Truthound Core can check a prepared structural distinct count once the operator has defined how documents are compared. [1, 2, 3]
A reported distinct collision
ClickHouse issue #129458 reports count(DISTINCT) merging two structurally different JSON documents on 26.10.2.103. The reporter provides a GROUP BY comparison and suggests hash encoding as a possible cause. Follow-up comments describe investigation, not a verified deployed correction. [1]
Change only object boundaries
Build a pair that changes only object boundaries inside an array. Preserve the raw JSON and a structure-aware reference identity, then compare distinct and control results. Define key order, numeric spelling, and NULL semantics before substituting strings for documents. [1, 3]
Two structures establish two documents
One x array contains {a:1} and {b:2,c:3}; the other contains {a:1,b:2} and {c:3}. Their scalar values overlap, but their object boundaries differ. Expected distinct documents=2 follows from that structure. A result of 2 passes; 1 collapses two reference identities and fails. [1, 3]
Trustworthy structural tokens
DistinctCountValidator in Truthound Core Python SDK 3.3.2 checks a column distinct count against min_count and max_count. Give it independently prepared structural tokens if those tokens represent the intended equality. It does not parse ClickHouse JSON internals or repair hashing. [1, 2, 3]
Keep parsing ahead of hashing
A structure-preserving parser can provide a comparison reference, while hashes remain an optimization. Verify that canonicalization itself retains object membership and array order. An independently different document must not receive the same token merely because field totals agree. [1, 3]
Pause the dependent metric
Hold distinct-based metrics when this control pair collapses. Two input rows alone are not the proof: the reference distinction comes from their different structures. Matching the count also leaves other equality edge cases to separate tests. [1, 3]
Application conditions
Prepare independent structure-preserving JSON identities and an equality policy.
Alternatives
Use a structural parser as reference rather than approving equality from hashes alone.
Limitations
The distinct-count check inspects prepared tokens and does not repair ClickHouse hashing.
Evidence
- Event checked
- Subject checked
- Modified
