Check quantile probabilities before interpreting Arrow tdigest NaN
Invalid probability requests and approximation error on valid requests require different checks.
Validate the configuration before aggregating
Check the probability domain before evaluating quantile output quality. A NaN returned for an invalid q can be mistaken for an ordinary missing aggregate. Retaining request settings as data separates invalid parameters from problems in the input observations before inspecting the results. [1, 2, 3]
Different responses to the same invalid probability
The PyArrow 26.0.0 report uses q=[0.5, 1.5]: quantile raises an error, while tdigest returns a value alongside NaN. It also demonstrates NaN from grouped hash_tdigest for an out-of-range q. The report explicitly leaves the contract for q=NaN unexamined, so it does not establish that behavior. [1]
Keep the requested probability list
Record requested q values and verify that every element is a finite probability between 0 and 1 before aggregation. Retain job_id and the algorithm name. If only some probabilities are valid, keeping their outputs while recording the whole request as successful obscures which requested percentiles were lost. [1, 3]
An upper-bound example and the complete input policy
With upper bound 1, q=0.5 passes and q=1.5 fails. These illustrative cases cover only that bound; the operational policy also needs lower bound 0 and finiteness. Truthound Core's BetweenValidator can apply the interval boundaries, with NaN exclusion declared as a separate check. Only after validating q should the application assess whether tdigest approximation error is acceptable. [1, 2, 3]
Application conditions
Prepare job_id, requested_q and algorithm settings from the original request or an independent control, aligned to the same run, file and key scope.
Alternatives
Use a wrapper that validates probability options first and evaluate approximation accuracy separately.
Limitations
A valid q does not guarantee acceptable tdigest approximation error. The source did not establish the q=NaN contract.
Evidence
- Event checked
- Subject checked
- Modified
