Separate successful Doris external-scan completion from data completeness
A successful Spark job can still contain a short read if cancellation or failure is exposed as normal end-of-stream.
A completion flag has limited meaning
Normal termination and data completeness must be independent release conditions. The existence of a table for Truthound to inspect does not establish that the original scope was fully read. Detecting a short read requires an independent reference for the same snapshot. [1, 2, 3]
A path that does not recheck failure state
The public Doris report describes checking queue status before blocking, then returning a normal-looking end after cancellation or failure occurs. A control shows a downstream Spark job succeeding despite missing rows. Distinguishing a legitimately empty scope from a truncated read is the central task. [1]
An illustrative fixed-scope comparison
If an independent reference establishes 5 rows in the inspected source snapshot, an extraction of 5 passes the count condition and 2 fails. A count collected at another time while the source changes is not a valid reference. Equal counts can also hide different rows, so follow the count check with a key-set comparison. [1, 3]
Prepare evidence before invoking Truthound
Fix the scan snapshot and expected key range, and compare the result with an independent source aggregate. Do not approve a short read as a normal empty result. Truthound Core's RowCountValidator can apply the approved count, but snapshot alignment and execution evidence must be supplied externally. It does not fix cancellation-status propagation. A second reader must inspect the same source point as well. [1, 2, 3]
Application conditions
Obtain independent source_rows and actual extracted_rows for the same snapshot_id and key_range, retaining execution completion state.
Alternatives
Compare keys and row counts using another reader on the same snapshot.
Limitations
A count difference across different source times is not necessarily failure evidence. Truthound does not repair external-scan cancellation propagation.
Evidence
- Event checked
- Subject checked
- Modified
