An all-NaN partial_fit batch is not permission to erase accumulated statistics
Treating a feature's lack of new observations as invalidation of earlier statistics can turn later finite inputs into NaN.
An empty batch does not erase earlier observations
No new observation is different from invalidating previous observations. Preserve each feature's accumulated support and state. In a stream, one feature can be temporarily empty while others continue arriving. Handing over only batch-level success can hide the loss of a particular column's statistics from the next analyst. [1, 2, 3]
A completed update with a downstream effect
scikit-learn issue 35143 reports that MinMaxScaler and MaxAbsScaler partial_fit can overwrite accumulated statistics with NaN when a later batch contains only NaN for a previously observed feature. It shows subsequent finite inputs transforming to NaN and contrasts a fit on the combined data. Read this as the reported environment and comparison, not a claim about every streaming preprocessor. [1]
Hand over feature-level update evidence
Mark batches containing empty features and reconcile prior valid statistics and observation counts before sharing transformed results. Record any time-window expiration or data-deletion policy first. A cumulative statistic that should retain earlier observations and a windowed statistic that must discard them do not share the same invariant. [1, 3]
When the maximum should remain 3
Define an illustrative first batch with valid values 1 and 3, followed by a batch entirely missing for that feature. Without observation deletion, the expected maximum remains 3. A post-update 3 passes; NaN fails because no new observation has been converted into no accumulated information. If the feature had always been missing, this expected value would not exist. NaN here denotes missing statistical state. [1, 3]
A conditional check does not repair the updater
ConditionalExpressionValidator in Truthound Core 3.3.2 applies then_expr only to rows selected by condition. A prepared table can select cases with no new valid observation but a valid prior statistic and check preservation. Specify null and NaN handling before connecting the table. The validator does not restore scaler state. Correct and retest the affected update path, and communicate the scope of any reused statistics. [1, 2, 3]
Application conditions
Prepare feature_id, batch_valid_count, previous_max and updated_max from raw input and approved references, preserving row identity.
Alternatives
Update incremental state while separating per-feature valid-observation counts from the approved forgetting policy.
Limitations
A data-deletion or time-window-expiration policy changes the conditions for retaining prior statistics.
Evidence
- Event checked
- Subject checked
- Modified
