Styxis
Company
Knowledge

A missing Logstash DLQ segment: check the next writer before deleting data

The impact of the same exception depends on its position in the writer lifecycle.

Lifecycle state behind an exception

The impact of the same exception depends on its position in the writer lifecycle. Check whether the next writer opened before deleting retained events. A Truthound liveness table needs both exception site and writer state; merging all such errors can stop a healthy pipeline or destroy recovery material. [1, 2, 3]

Different effects at two call sites

Logstash #19377 reports a race between DLQ reader deletion and Files.size in 9.4.2, potentially leaving a closed writer and a pipeline cascade after upgrading from 9.0.8. The author distinguishes the fatal path inside finalize from a harmless call after the next writer has opened. The report is not our reproduction of either production cluster. [1]

Preserve recovery material first

Join finalize phase, nextWriter reachability and subsequent write acceptance. Hold recovery completion if the writer remains closed. Preserve source and DLQ files first and use only an approved isolated probe; do not mix invented test records into user-log completeness counts. [1, 3]

The one condition a probe checks

The illustrative finalize contract requires an open next writer accepting an approved probe. Acceptance 1 meets only that liveness condition; a closed writer giving 0 fails. Check probe content separately, and do not infer historical blackout losses from this flag. [1, 3]

Compare prepared phase traces

Truthound Core Python SDK 3.3.2 can compare prepared required_post_finalize_write_ok and post_finalize_write_ok with ColumnPairValidator using relationship="==". The operator supplies phase traces and safe probe outcomes. The SDK does not delete files or reopen writers, and a liveness check does not establish recovery of historical logs. [1, 2, 3]

Different responses to the same exception

Assess retention and storage costs before selecting clean_consumed or retain.age workarounds. A standalone failure after nextWriter may leave a live writer and does not justify the same deletion-and-restart response. Keep correction testing and source reconciliation for the blackout interval as separate completion items. [1, 3]

Application conditions

Preserve source and DLQ material, use an approved isolated probe and distinguish fatal from harmless call sites.

Alternatives

Review retention and capacity costs of the cleanup workaround and test a supported correction.

Limitations

  • The SDK does not delete segments or reopen writers; probe success does not prove historical recovery.

Evidence

Event checked
Subject checked
Modified