Styxis
Company
Knowledge

Check quantile probabilities before interpreting Arrow tdigest NaN

Invalid probability requests and approximation error on valid requests require different checks.

Validate the configuration before aggregating

Check the probability domain before evaluating quantile output quality. A NaN returned for an invalid q can be mistaken for an ordinary missing aggregate. Retaining request settings as data separates invalid parameters from problems in the input observations before inspecting the results. [1, 2, 3]

Different responses to the same invalid probability

The PyArrow 26.0.0 report uses q=[0.5, 1.5]: quantile raises an error, while tdigest returns a value alongside NaN. It also demonstrates NaN from grouped hash_tdigest for an out-of-range q. The report explicitly leaves the contract for q=NaN unexamined, so it does not establish that behavior. [1]

Keep the requested probability list

Record requested q values and verify that every element is a finite probability between 0 and 1 before aggregation. Retain job_id and the algorithm name. If only some probabilities are valid, keeping their outputs while recording the whole request as successful obscures which requested percentiles were lost. [1, 3]

An upper-bound example and the complete input policy

With upper bound 1, q=0.5 passes and q=1.5 fails. These illustrative cases cover only that bound; the operational policy also needs lower bound 0 and finiteness. Truthound Core's BetweenValidator can apply the interval boundaries, with NaN exclusion declared as a separate check. Only after validating q should the application assess whether tdigest approximation error is acceptable. [1, 2, 3]

Application conditions

Prepare job_id, requested_q and algorithm settings from the original request or an independent control, aligned to the same run, file and key scope.

Alternatives

Use a wrapper that validates probability options first and evaluate approximation accuracy separately.

Limitations

  • A valid q does not guarantee acceptable tdigest approximation error. The source did not establish the q=NaN contract.

Evidence

Event checked
Subject checked
Modified