A machine-learning experiment is not rigorous because it uses a complex model. Rigour comes from a question that matters, a dataset that represents the intended use, an evaluation design that prevents information leakage and conclusions that remain within the evidence.
Many apparently strong results fail under review because the comparison is unfair, the test set influenced development, the metric hides important errors or uncertainty is not reported. The following framework makes the evaluation auditable.
1. Define the scientific question before the model
State whether the study is trying to predict, explain, detect, rank, generate, cluster or estimate an effect. Define the unit of prediction, target population, deployment context and consequence of different errors.
A research contribution may be a new method, a stronger evaluation, a dataset, an application, an empirical finding or a reproducibility result. Make the contribution explicit so the experiment tests it directly.
2. Audit the data-generating process
Document where observations came from, how labels were produced, which exclusions occurred and whether repeated measurements belong to the same person, device, location or period. These relationships determine how data must be split.
- Check missingness, duplicates, label noise and class imbalance.
- Identify sensitive attributes and plausible sources of systematic bias.
- Record preprocessing decisions and fit them only on training data.
- Keep a data dictionary and provenance record.
3. Design splits that represent generalisation
Random row-level splitting is inappropriate when related observations cross partitions. Use grouped splits for repeated entities, temporal splits for future prediction and site-based splits when generalisation to new institutions matters.
Use validation data for model and hyperparameter decisions. Preserve the test set for the final, pre-specified evaluation. Repeatedly checking test performance turns the test set into another validation set.
4. Choose meaningful baselines
Compare against simple heuristics, conventional statistical models and credible prior methods. A new system that beats a weak baseline does not demonstrate a useful advance. Keep data access, tuning effort and evaluation conditions comparable across methods.
Ablation studies should isolate which components create the observed improvement. If several changes are introduced together, their individual contribution remains unknown.
5. Match metrics to the decision
| Context | Useful considerations | Common mistake |
|---|---|---|
| Imbalanced classification | Precision, recall, PR-AUC, class-specific errors | Reporting accuracy alone |
| Risk prediction | Discrimination, calibration and decision thresholds | Assuming high AUC means reliable probabilities |
| Regression | Error distribution, MAE/RMSE, scale and practical tolerance | Reporting one aggregate without residual analysis |
| Ranking or retrieval | Cut-off-specific ranking metrics and user relevance | Using a metric disconnected from retrieval depth |
| Generative systems | Task measures, human evaluation, safety and failure categories | Relying on one automated similarity score |
6. Report uncertainty and variation
Single-run point estimates hide instability. Where appropriate, report confidence intervals, repeated splits, cross-validation distributions or variation across random seeds. If models are compared, use a comparison method suited to the dependence in the evaluation data.
Distinguish statistical detectability from practical importance. A very small improvement can be statistically detectable in a large test set yet irrelevant in practice.
7. Analyse errors, subgroups and robustness
Aggregate performance is only the beginning. Inspect error types, difficult cases and performance across meaningful subgroups. Test sensitivity to distribution shift, missing features, noise or plausible changes in preprocessing when these conditions matter to the intended use.
For high-stakes applications, examine calibration, failure costs, fairness, human oversight and whether the system should be used at all. Responsible research includes identifying situations in which the model is unreliable.
8. Make reproduction possible
Report data access and exclusions, split logic, preprocessing, model architecture, hyperparameters, tuning budget, software versions, hardware where relevant, random seeds and exact metric definitions. Share code, configuration and model artefacts when permissions and ethics allow.
Finally, align the conclusion with the evaluation domain. Performance on one retrospective dataset does not establish clinical, operational or population-wide effectiveness. State what was demonstrated, what remains uncertain and which future test would reduce that uncertainty.
We can review the research question, split logic, baselines, metrics and reporting plan before expensive experimentation begins.
Discuss ML research support