Styxis
Company
Knowledge

Adjusted balanced accuracy on a single-class fold: keep undefined states separate from scores

Replacing an undefined adjusted score with an arbitrary finite value changes both metric meaning and the population being evaluated.

Check the domain before filling a number

When sharing group-level evaluation, do not replace an undefined metric with either excellent or worst performance. Report its applicable domain. Small or imbalanced folds can retain only one effective class. Producing a value and having a valid basis for comparing models are separate questions. [1, 2, 3]

An adjustment path with denominator 0

scikit-learn issue 34889 demonstrates NaN or negative infinity from balanced_accuracy_score with adjusted=True when one class survives the internal missing-value filter. It explains that chance becomes 1 and the adjustment divides by 1-chance. The report does not settle the correct replacement score. This article proposes an evaluation-handoff policy that keeps undefined states visible. [1]

Keep omitted folds in the scope report

Record effective class counts and whether chance adjustment applies to each fold. Separate undefined folds from numerical averaging while reporting coverage. Calling an average over the remaining folds an average for all targets hides a changed evaluation population. Preserve sample counts and exclusion reasons, leaving metric redefinition or split changes to the evaluation owner. [1, 3]

A finite value is not automatically valid

An illustrative fold with 1 effective class has adjustment denominator 0. Set expected status to undefined; retaining undefined preserves the domain. Inserting an arbitrary number and labeling it finite_score fails by erasing that state. A finite-value validator can accept the replacement number and therefore cannot resolve this problem. The state names are proposed ledger representations, not API return enums. [1, 3]

Use Truthound to check the state's conditions

ConditionalExpressionValidator in Truthound Core 3.3.2 can conditionally check evaluation state against a prepared effective-class count. Compute that count and adjustment applicability correctly outside the validator and specify missing-value handling. Truthound does not choose a suitable replacement score or a fair evaluation split. Disclose the undefined scope and keep any recalculated result distinct from the original in the handoff. [1, 2, 3]

Application conditions

Prepare fold_id, effective_class_count, adjusted_metric_defined and score from raw data and approved references, preserving row identity.

Alternatives

Redesign evaluation splits and class-support conditions, or use a different metric defined for the intended purpose.

Limitations

  • Silently excluding undefined folds changes the shared evaluation population. Report both exclusion scope and sample counts.

Evidence

Event checked
Subject checked
Modified