Beam Parquet window mixing: readable files still need membership checks
Compare event identity with its assigned window across interleaving and flush boundaries.
Readable files with the wrong membership
A readable file does not establish that an event stayed in its assigned window. Check whether the last observed window incorrectly owns rows from earlier windows. Supply Truthound with file paths and independently assigned event-time windows together; a correct total count can still hide incorrect per-window results. [1, 2, 3]
Window buffers and pane metadata differ
Beam #40284 reports a Python Parquet buffer emitting rows from different windows under the last window. The follow-up description separates per-window buffering from pane-info propagation. Issue closure does not establish a particular released version, and a window-buffer change must not be described as preserving pane metadata too. [1]
Interleave input and flush midway
Interleave rows from two windows within one bundle and compare event-id sets before and after a mid-bundle flush. Withhold aggregates for files whose membership changed. Keep the source event ids and event-time assignments independently, then join writer metadata and paths. A control grouping all rows by window does not exercise the shared-buffer boundary. [1, 3]
A belongs in a W1 file
The illustrative event A is assigned to fixed window W1, so its file membership must also be W1. W1 meets this condition; a W2 file fails even if the total element count is unchanged. Duplicates and pane information need separate checks. This is a defined input, not a Beam execution result. [1, 3]
Establish window assignment externally
Truthound Core Python SDK 3.3.2 can compare prepared assigned_window_id and observed_window_id using ColumnPairValidator with relationship="==". The operator supplies keys and assignments and exposes missing events through an external join. The SDK does not repair writer memory buffers. Legitimate multiple memberships in sliding windows require a separate relationship table. [1, 2, 3]
Retain the regression conditions
Test interleaved windows and mid-bundle flushes separately on the exact target version. A single global-window job has a different condition. Even correct membership does not establish pane preservation or release adoption. Hold time-bucketed aggregates for affected files until the relevant reconciliation is complete. [1, 3]
Application conditions
Preserve event-time assignments and writer window metadata, allowing legitimate multiple memberships for sliding windows.
Alternatives
Confirm the version with per-window buffering and separately test interleaved windows and mid-bundle flushes.
Limitations
Truthound does not repair writer buffers; correct window membership does not establish pane preservation or release adoption.
Evidence
- Event checked
- Subject checked
- Modified
