Reproducible experiments: baselines, fair comparisons and failure audits

Audience and output

For readers able to run baselines and manage scripts and splits. Produce a main-result reproduction package, fairness statement, variability report and failure log. This platform-designed exercise concerns computational research; wet-lab work also needs domain-specific protocols and controls.

The Turing Way supports environment and data management; the NeurIPS checklist helps audit reporting. Rerunning code, reproducing a reported result and establishing a valid conclusion are separate achievements.

Freeze the claim and materials

Suppose a paper claims improved small-sample classification. Specify paper version, target table, metrics, data licence, splits, preprocessing, model versions, hyperparameters, environment and budget. Record missing information instead of silently filling gaps and claiming identical conditions.

Maintain a data manifest with sources, versions or checksums, permissions, retrieval commands and exclusions. Keep raw data read-only and script transformations. Use controlled access or synthetic examples when sensitive data cannot be shared.

Make comparisons fair

Fit learned preprocessing on training data. Prevent participants or other entities leaking across splits. Use validation for selection and keep test data out of tuning. Choose splits to reflect intended generalization.

Report search spaces, tuning budgets, early stopping, external data, pretraining and compute for both methods and baselines. A tuned new method against a default baseline does not establish a fair superiority claim. Disclose and limit resource-mismatched comparisons.

Identify sources of variability

Specify repetitions and rationale in advance and retain seeds. Seed variation measures training randomness, not sampling uncertainty; cross-validation folds can also be dependent. State the statistical unit and interval calculation. Repeated runs do not create independent datasets.

Ablate a component under comparable budgets, then examine subgroups or distribution shifts. Ablation tests a mechanism under stated conditions, without eliminating every alternative explanation.

Audit discrepancies

Trace disagreements through data, preprocessing, implementation, training, environment and evaluation. Separate verified causes, hypotheses and unknowns. Do not select the best seed or repeatedly change the test set until results match.

Review criteria

• Supply commands from permitted data retrieval to the main table, with hardware, time and cost assumptions.
• Include environment locks, versions, configurations, seeds, manifest and raw outputs.
• Have another person run a clean environment; record success, tolerances and manual steps.
• Compare original and reproduced values and discuss fairness, variability, failures and claim boundaries.

Open reproducibility resources; for new quantitative studies, also justify the design's information target.

Advanced practice and review evidence

Build a teaching example that deliberately puts entities in both training and test sets. Compare record-level and entity-level splitting and explain the generalization target of each. Fit preprocessing on training only; record tuning budget, external data and stopping rules for baseline and proposed methods.

Deliver an input manifest, environment lock, commands, main table, tolerances and failure log. Rerun cleanly and document manual steps. Seeds represent training variation, not independent studies. Transfer: only one run is affordable; limit uncertainty claims and record missing checks instead of declaring validation.

Open practice, AI review and work export.