Adjusted balanced accuracy on a single-class fold: keep undefined states separate from scores
Replacing an undefined adjusted score with an arbitrary finite value changes both metric meaning and the population being evaluated.
Check the domain before filling a number
When sharing group-level evaluation, do not replace an undefined metric with either excellent or worst performance. Report its applicable domain. Small or imbalanced folds can retain only one effective class. Producing a value and having a valid basis for comparing models are separate questions. [1, 2, 3]
An adjustment path with denominator 0
scikit-learn issue 34889 demonstrates NaN or negative infinity from balanced_accuracy_score with adjusted=True when one class survives the internal missing-value filter. It explains that chance becomes 1 and the adjustment divides by 1-chance. The report does not settle the correct replacement score. This article proposes an evaluation-handoff policy that keeps undefined states visible. [1]
Keep omitted folds in the scope report
Record effective class counts and whether chance adjustment applies to each fold. Separate undefined folds from numerical averaging while reporting coverage. Calling an average over the remaining folds an average for all targets hides a changed evaluation population. Preserve sample counts and exclusion reasons, leaving metric redefinition or split changes to the evaluation owner. [1, 3]
A finite value is not automatically valid
An illustrative fold with 1 effective class has adjustment denominator 0. Set expected status to undefined; retaining undefined preserves the domain. Inserting an arbitrary number and labeling it finite_score fails by erasing that state. A finite-value validator can accept the replacement number and therefore cannot resolve this problem. The state names are proposed ledger representations, not API return enums. [1, 3]
Use Truthound to check the state's conditions
ConditionalExpressionValidator in Truthound Core 3.3.2 can conditionally check evaluation state against a prepared effective-class count. Compute that count and adjustment applicability correctly outside the validator and specify missing-value handling. Truthound does not choose a suitable replacement score or a fair evaluation split. Disclose the undefined scope and keep any recalculated result distinct from the original in the handoff. [1, 2, 3]
Application conditions
Prepare fold_id, effective_class_count, adjusted_metric_defined and score from raw data and approved references, preserving row identity.
Alternatives
Redesign evaluation splits and class-support conditions, or use a different metric defined for the intended purpose.
Limitations
Silently excluding undefined folds changes the shared evaluation population. Report both exclusion scope and sample counts.
Evidence
- Event checked
- Subject checked
- Modified
