Guidelines for Machine Learning Experimental Projects
1) Define the problem and hypothesis
- State the task type (classification, regression, ranking, generation, time series, etc.).
- Specify the target variable(s) and input modalities.
- Formulate a testable hypothesis (e.g., “X feature set will improve PR-AUC over baseline by ≥3%”).
- Identify constraints: compute budget, data availability, latency, interpretability, and deadlines.
- Declare success criteria upfront (primary metric, acceptable thresholds).
2) Review prior work and set baselines
- Summarize 3–5 relevant papers or implementations and their metrics on comparable data.
- Choose at least two baselines:
- A trivial baseline (e.g., majority class, mean regressor).
- A strong, simple baseline (e.g., regularized linear/logistic model, shallow tree).
- Justify why each baseline is appropriate for your data and metrics.
3) Plan evaluation and data splits
- Keep a final test set untouched until all modeling decisions are complete.
- Choose a split strategy that matches data structure:
- i.i.d. data: train/validation/test or K-fold cross-validation.
- Imbalanced classification: use stratified splits.
- Grouped data (e.g., multiple records per user): use group-aware splits (GroupKFold).
- Time series: use temporal splits (walk-forward/rolling-origin validation). Do not shuffle across time.
- For small datasets or extensive hyperparameter tuning, use nested cross-validation to avoid optimistic bias.
- Document the exact split method, random seeds, and group/time boundaries.
4) Govern data ethically and legally
- Verify licenses and terms of use for all datasets and pretrained models.
- Remove or protect personally identifiable information (PII). Minimize sensitive attributes unless explicitly needed for fairness analysis.
- If human subjects or sensitive data are involved, follow your institution’s review/approval processes.
- Record data provenance (source, date, version, filters) and maintain a data card.
5) Audit and understand your data
- Compute descriptive statistics: counts, missingness, ranges, class balance.
- Visualize distributions and correlations; identify outliers and duplicates.
- Check for target leakage (features that directly or indirectly reveal the label).
- Define and document handling of missing data (impute within training folds only).
6) Build a leakage-safe preprocessing pipeline
- Fit scalers/encoders/imputers only on training data within each fold/split.
- Apply the fitted transformers to validation/test data without refitting.
- For text/images/audio, apply augmentation only on training data; keep validation/test clean.
- Save pipeline objects and code so the exact steps can be reproduced.
7) Choose models that match the problem and constraints
- Start with interpretable or efficient models; escalate complexity only if needed.
- Justify each model family (e.g., tree ensembles for tabular; CNNs/Transformers for images/text; sequence models for time series).
- Note deployment constraints (e.g., model size, inference time).
8) Design controlled experiments
- Change one factor at a time when possible (model, feature set, loss, augmentation).
- Predefine the hyperparameter search space and budget to avoid “fishing.”
- Use identical data splits and preprocessing across model comparisons.
- Run multiple random seeds for stochastic models; report mean and standard deviation.
9) Tune hyperparameters responsibly
- Use validation data or inner cross-validation only; do not use test data for tuning.
- Pick a search strategy appropriate to budget: grid (small), random (broad), Bayesian/early-stopping methods (efficient).
- Record: search space, sampling strategy, number of trials, time per trial, early-stopping criteria.
10) Select appropriate metrics
- Classification: accuracy (balanced data), precision/recall/F1, ROC-AUC; prefer PR-AUC when classes are highly imbalanced.
- Regression: MAE and RMSE; report both when possible; include R^2 with caution on small samples.
- Ranking/retrieval: MAP, NDCG@k.
- Calibration (if probabilities matter): reliability diagram, Brier score, Expected Calibration Error.
- Define how thresholds are chosen (on validation only) and keep them fixed on the test set.
11) Perform statistical and uncertainty analysis
- Report confidence intervals (e.g., bootstrap for many metrics).
- For paired model comparisons on the same test set:
- Use McNemar’s test for 0/1 classification errors.
- Use DeLong’s test for differences in ROC-AUC.
- Use bootstrap for PR-AUC, MAE/RMSE, or when assumptions are unclear.
- Adjust for multiple comparisons when testing many models (e.g., control false discovery rate).
12) Conduct error analysis
- Inspect confusion matrices and per-class metrics.
- Review representative false positives/negatives or large-error residuals.
- Analyze performance by meaningful subgroups (e.g., demographics, device types); look for disparate impact.
- Trace errors back to data quality issues and feature/model limitations.
13) Run ablations and robustness checks
- Ablate features/modules to identify what truly contributes to performance.
- Stress test with distribution shifts, noise, corruptions, or realistic perturbations.
- Evaluate out-of-distribution performance when relevant; clearly label it.
- Test stability across seeds, data subsamples, and minor preprocessing changes.
14) Address fairness, privacy, and safety
- If sensitive attributes are available, report group-wise metrics and gaps.
- Consider simple mitigations (reweighting, threshold adjustments) and discuss trade-offs.
- Avoid releasing models that could be misused; document intended use and limitations.
- Do not publish sensitive data or inadvertently deanonymize individuals.
15) Ensure reproducibility
- Fix and record random seeds for all libraries; run multiple seeds for results.
- Pin library versions; save an environment file (e.g., requirements.txt, conda env, Dockerfile).
- Enable deterministic operations when feasible and note hardware specifics (CPU/GPU, model of GPU).
- Log data hashes or dataset versions; store train/val/test indices.
- Use an experiment tracker (e.g., MLflow, Weights & Biases) or a clear logging system.
16) Manage compute and efficiency
- Monitor and record training time, memory, and hardware used.
- Use early stopping and learning-rate schedules where appropriate.
- Right-size models; prefer techniques like ensembling only if the benefit justifies the cost.
- Consider mixed precision if supported; verify numerical stability.
17) Document thoroughly
- Maintain a README with how to reproduce results end-to-end.
- Provide a clear repo structure (data/, src/, configs/, experiments/, notebooks/, reports/).
- Include a model card (intended use, metrics, data, limitations) and a datasheet/data card.
- Document all negative results and failed approaches; explain what was learned.
18) Report results transparently
- Present validation and test results separately; keep the test set final and untouched until the end.
- Show averages and variability (mean ± std or confidence intervals) across seeds/folds.
- Include learning curves and key plots (PR/ROC, calibration, residuals).
- State compute budget, number of trials, and any early stopping or pruning.
- Avoid cherry-picking; include all pre-specified comparisons.
19) Collaborate and control versions
- Use version control (Git) with meaningful commit messages.
- Track data and model artifacts with checksums or data versioning tools where possible.
- Assign code reviews and use issue tracking for tasks and bugs.
20) Deliverables checklist (adapt to your course/project)
- Proposal: problem statement, hypothesis, metrics, split plan, risks.
- Midpoint update: baselines, initial results, revised plan.
- Final package: code, environment, data access instructions, trained models (if allowed), report with full methodology, analysis, and limitations.
Common pitfalls to avoid
- Tuning on the test set or peeking at test performance during development.
- Data leakage via preprocessing fit on full data or target-dependent features.
- Reporting single-run results for stochastic training without variability.
- Comparing models on different splits or with different preprocessing.
- Ignoring subgroup performance and model calibration.
- Overclaiming generalization without external or temporally separated validation.
Use this as a step-by-step checklist. At each stage, write down decisions, justify them, and keep artifacts so your work is transparent, reproducible, and educationally valuable.
