Styxis
Company
Knowledge

Read final Pinot metadata before accepting an empty Trino split

Exclude a Pinot split from successful aggregation when final metadata reports an error, even if its data blocks are empty. Keep missing metadata distinct from a valid empty result so a low count cannot conceal a missing segment.

Empty still needs completion evidence

An empty split is not a normal absence until its final metadata has been checked for server errors. Truthound Core can inspect an operator-prepared approval flag conditioned on that metadata evidence. [1, 2, 3]

A final metadata exception report

Trino issue #31888 reports the Pinot streaming gRPC connector stopping at METADATA without parsing exceptions. A moved segment can then yield 0 rows without an error. A follow-up links a PR handling that block; local rollout is a separate question. [1]

Preserve unknown completion

Collect split ID, data-block count, final metadata receipt, exception code, and approval state. If SERVER_SEGMENT_MISSING is present, exclude the result from successful aggregates. Missing final metadata itself must not be converted to a known error-free completion. [1, 3]

Error 235 blocks approval

Define an illustrative split whose final metadata contains SERVER_SEGMENT_MISSING=235. Expected success approval=false regardless of result count. false passes; true fails because an unavailable segment would be counted as a legitimate empty range. [1, 3]

Capture gRPC outside the SDK

ConditionalExpressionValidator in Truthound Core Python SDK 3.3.2 can apply the prepared error condition to a success flag. The operator captures gRPC evidence and defines missing-metadata handling. The SDK does not collect responses or relocate Pinot segments. [1, 2, 3]

Replan the same segment range

Replan the same segment range through a supported connector path that processes final metadata. A retry with a different snapshot is useful operationally but cannot silently replace the original reference boundary. [1, 3]

Separate empty error and unknown

Keep normal empty, server exception, and unknown completion as distinct states. That distinction makes intermittent low counts actionable without inventing an expected row count for a moving realtime segment. [1, 3]

Application conditions

Prepare per-split final-metadata receipt, exceptions, and approval.

Alternatives

Replan the same segment range through a final-metadata-aware path.

Limitations

  • Do not fill missing final metadata as error-free completion.

Evidence

Event checked
Subject checked
Modified