Identical GCS CSV rows: separate payload duplication from record identity
Use object generation and record positions to preserve distinct observations, then reconcile their sink identities.
Identical content can still be two records
Separate object identity from the identities of records inside it. Two identical payloads may be distinct observations, so a payload fingerprint must not silently collapse them. Decide the observation unit before using Truthound; a successful object-level status cannot establish that every CSV record survived. [1, 2, 3]
IDs collide after codec decoding
Filebeat GCS #53659 reports multiple CSV records receiving the same document id and offset, leaving only the first record. It describes evtOffset not advancing, producing the same objectID and log.offset 0. Azure is mentioned as a related possibility, not an equivalent measured reproduction. The proposed workaround still needs a business-semantics assessment. [1]
Keep the original record positions
Build a source inventory with object generation and record ordinal, verify that each record reaches a distinct sink ID, then reprocess the missing scope. Freeze the generation and retain the codec-defined record boundaries. A physical line number is not automatically a record ordinal for multiline records; also retain create and overwrite outcomes. [1, 3]
Three positions but two payloads
The illustrative CSV has 3 record positions, with identical content in the first two. The contract preserves all 3 observations, so 3 sink identities are expected, although there are only 2 distinct payloads. A distinct-ID count of 3 meets only the count condition and still requires position mapping. A count of 1 fails; fingerprint output of 2 also falls short. This is a defined fixture, not a GCS reread result. [1, 3]
Count identities and inspect correspondence
Truthound Core Python SDK 3.3.2 can check the fixture’s distinct sink-ID count using DistinctCountValidator with min_count=3 and max_count=3. Calculate source-position-to-sink-ID correspondence separately. The SDK does not repair evtOffset or reread the object, and the operator must establish the complete input-table scope. [1, 2, 3]
Before choosing a fingerprint
A fingerprint workaround may fit retransmission deduplication. If two equal payloads represent independent observations, use an identity design including object generation and record ordinal. After selecting that policy, reprocess only the missing scope and reconcile duplicates among already landed records. [1, 3]
Application conditions
Use stable record boundaries and object generation, first deciding whether identical payloads are independent observations or business duplicates.
Alternatives
Assess repeated-payload meaning before testing the fingerprint workaround, or include record ordinals in identity design.
Limitations
Truthound does not repair Filebeat evtOffset or reread GCS objects; distinct-ID counts alone do not establish per-record correspondence.
Evidence
- Event checked
- Subject checked
- Modified
