Bigtable RESUME_OR_NEW: count partition consumers inside one job
Job count is different from partition-consumer ownership.
Ownership beyond the job count
Job count is different from partition-consumer ownership. Check whether repeated initialization inside one execution creates another consumer chain. A Truthound table containing only job counts would hide that duplication; record overlapping ownership separately from duplicated business effects. [1, 2, 3]
Conditions of the persistent-duplicate report
Beam #39970 reports persistent duplicate consumption with Bigtable RESUME_OR_NEW on Java SDK 2.75.0 despite one job and one worker. The report distinguishes this from a short checkpoint overlap. Reinitialization, existing metadata and same-UUID locking are its suspected explanation; proposed fixes are not evidence of our integration test or a shipped repair. [1]
Decompose initialization
Compare active consumers per partition before and after initialization repeats within the same job. Isolate intervals with duplicate mutation effects. Retain chain id, partition, lock generation and execution id, distinguishing same-execution reinitialization from an authorized new-job resume. [1, 3]
One chain for partition P
The illustrative job J allows one active chain for partition P. An observed count of 1 after reinitialization meets the ownership-count condition; 2 fails. The comparison alone does not establish source coverage or exactly-once effects from the single remaining chain. [1, 3]
Compare counts at the same time
Truthound Core Python SDK 3.3.2 can compare prepared approved_consumer_chains and active_consumer_chains with ColumnPairValidator using relationship="==". The operator aligns observation time and lock generation. The SDK does not change Bigtable metadata locking or Dataflow initialization. [1, 2, 3]
The cost of blocking resume
FAIL_IF_EXISTS is a reported workaround, but it also blocks legitimate reuse of metadata by a later job. Record that cost and test both repeated initialization within one execution and a new execution’s resume on the target version. A short authorized handoff overlap needs its own ownership rule. [1, 3]
Application conditions
Capture chain identities, partition and lock generation, distinguishing reinitialization from a legitimate new-job resume.
Alternatives
Test FAIL_IF_EXISTS with its legitimate-resume cost, or validate a supported correction.
Limitations
The SDK does not repair locking or initialization; one chain does not prove complete mutations.
Evidence
- Event checked
- Subject checked
- Modified
