B@20 after a Flink checkpoint: a watermark can pass the remaining buffer
A completed checkpoint buffer is different from the current buffer receiving later records.
The next buffer is still pending
A completed checkpoint buffer is different from the current buffer receiving later records. Check whether the watermark passed an event that has not been emitted. Truthound needs both buffer observations and watermark state to identify premature finalization; a checkpoint-complete log alone cannot establish safety for all pending data. [1, 2, 3]
The sequence checked by the harness
Beam #39991 reports a source-method harness where completing checkpoint 1 emits A@10 but leaves B@20 buffered while the watermark advances to 100. The report explicitly says the proposed Flink JUnit regression and a full integration test were not run. Keep those levels of evidence distinct. [1]
Align observations within the operator
Compare each checkpoint’s remaining minimum event time with its output watermark. Hold downstream publication that treats un-emitted B as already late. Join checkpoint and buffer id within the same operator and retain the minimum timestamp of pending records, avoiding observations taken after the buffer became empty. [1, 3]
A watermark beyond 20
In the illustrative state after checkpoint 1, B@20 is still buffered. A conservative output watermark must be at most 20, so the expected watermark-past-buffer flag is 0. Watermark 100 gives flag 1 and fails. Neither event counts nor monotonic watermark progress replace this preservation condition. [1, 3]
Prepare the overrun flag
Truthound Core Python SDK 3.3.2 can compare prepared permitted_past_buffer_flag and watermark_past_buffer_flag using ColumnPairValidator with relationship="==". The operator derives the flag from observations at the same point in the operator trace and isolates missing traces. The SDK does not restore Flink holds or events discarded downstream. [1, 2, 3]
When the hold may be released
Test new buffers between checkpoints and bundle-completion conditions on the target runner. Once B is emitted or the buffer is empty, the earlier 20 bound no longer applies. Determine recovery only after reconciling event membership and the downstream reprocessing scope. [1, 3]
Application conditions
Retain checkpoint and pending-buffer timestamps for the same operator and derive watermark-overrun flags externally.
Alternatives
Run an integration test covering a new buffer between checkpoints and bundle completion on the corrected runner.
Limitations
The SDK does not restore watermark holds or discarded events; source-method harness evidence is not full runner integration evidence.
Evidence
- Event checked
- Subject checked
- Modified
