Overfitting vs. Underfitting in Machine Learning
1) Definitions
- Overfitting: The model captures spurious patterns and noise specific to the training data, achieving very low training error but poor generalization (high validation/test error). It is characterized by high variance.
- Underfitting: The model is too simple or too constrained to capture the underlying signal, leading to high error on both training and validation sets. It is characterized by high bias.
2) Diagnostic Patterns
- Error behavior
- Overfitting: training error ≪ validation error; gap persists even with more training.
- Underfitting: training error is high and close to validation error; both plateau at poor performance.
- Learning curves (error vs. number of training samples)
- Overfitting: training error low; validation error high but decreases as more data is added; a large train–validation gap.
- Underfitting: both training and validation errors high and converge to a similar value; adding data yields limited improvement.
- Bias–variance perspective (for squared loss)
- Expected prediction error = bias^2 + variance + irreducible noise.
- Overfitting: low bias, high variance.
- Underfitting: high bias, low variance.
- Residuals
- Overfitting: residuals on training data near zero; on validation, structured residuals appear.
- Underfitting: residuals show clear structure on both training and validation (missed pattern).
3) Common Causes
- Overfitting
- Model capacity too high (deep trees, high-degree polynomials, large neural nets without regularization).
- Insufficient data or high label noise.
- Excessive training epochs without regularization/early stopping.
- Data leakage (test information influencing training or feature construction).
- High-dimensional feature space relative to sample size.
- Underfitting
- Model capacity too low (linear model for nonlinear relationship, shallow trees, overly strong regularization).
- Inadequate features (missing interactions/nonlinear transforms).
- Too little training (early stopping too soon, low learning rate without enough steps).
- Overly aggressive regularization or constraints.
4) Mitigation Strategies
- To reduce overfitting (reduce variance, improve generalization)
- Regularization: L2/L1 penalties, Elastic Net; weight decay; dropout; label smoothing.
- Model simplification: shallower trees; pruning; fewer parameters; lower polynomial degree.
- Early stopping: monitor validation metric and stop before overfitting increases.
- Data strategies: collect more data; data augmentation (images, text, audio); noise injection; mixup/cutmix (vision).
- Robust evaluation: stratified K-fold cross-validation; avoid leakage; proper temporal splits for time series.
- Feature selection/engineering: remove leaky or high-cardinality sparse features; target encoding with CV.
- Ensembling: bagging/Random Forest (reduces variance); snapshot ensembles; dropout ensembles.
- Calibration and constraints: monotonic constraints, causal/physics-informed features to restrict hypothesis space.
- To reduce underfitting (reduce bias, increase capacity)
- Increase model capacity: add features, interactions, basis expansions (kernels, splines), deeper networks, deeper trees.
- Reduce regularization strength; relax constraints.
- Feature engineering: nonlinear transforms, domain-specific features.
- Train longer or with improved optimization (learning rate schedules, better initialization).
- Use more expressive models (e.g., gradient boosting vs. linear regression).
5) Practical Workflow for Detection and Control
- Establish a sound validation scheme (stratified K-fold or time-aware splits); keep a final hold-out test set.
- Plot learning curves: training and validation metrics vs. number of samples and vs. epochs.
- Compare model families and regularization paths:
- Linear → polynomial → kernelized or tree-based methods; tune regularization (alpha, C, lambda) and complexity (tree depth, n_estimators).
- Monitor the train–validation gap:
- Large gap suggests overfitting → add regularization, simplify, add data/augmentation.
- Small but high errors suggest underfitting → increase capacity, features, or training time.
- Check for data leakage and distribution shift; ensure consistent preprocessing fitted only on training data.
- Use nested cross-validation for unbiased model selection when hyperparameter tuning is extensive.
6) Notes and Caveats
- In overparameterized regimes (e.g., very large neural networks), test error can exhibit “double descent.” Despite this, the operational diagnostics above (train–validation gap, learning curves, and careful validation) remain effective.
- Class imbalance and noisy labels can mimic overfitting symptoms; use appropriate metrics (e.g., AUC, F1, PR-AUC) and consider robust loss functions.
Summary
- Overfitting: low train error, high validation error, high variance. Fix with regularization, simplification, early stopping, more/better data, and robust validation.
- Underfitting: high train and validation error, high bias. Fix with more expressive models, richer features, weaker regularization, and sufficient training.
