Analyze, check models and reproduce results

Freeze the claim, unit and analysis target

Record paper version, target table, sampling unit, processing, metrics and tolerances. Repeated runs measure computational variation, not new samples. In prediction, split by the deployment unit and fit learned preprocessing only on training data. Do not repeatedly select on the test set. For explanation, define the estimand and identification assumptions before modeling.

Check the generative process

Compare data with what the model can generate through prior/posterior prediction or residual patterns, then run justified sensitivity checks. Computational convergence does not validate interpretation, and unfair tuning budgets do not establish superiority. Qualitative work needs a coding–theme–interpretation chain, negative cases and reflexivity rather than universally imposed agreement scores.

Audit AI-generated analysis

Hand-check a small example before auditing units, joins, missingness, leakage and seeds. Preserve environment locks, configurations, input checks and failure logs. Rerun in a clean environment with numerical tolerances. Without original data, label the result partial replication or an alternative check. AI's error explanations are hypotheses too.

Worked example · fictional

From a problem to inspectable work

Fictional: repeated answers from each student are split across training and test, producing high accuracy.

1. For predicting new students, cross-split identities create leakage.
2. Split by student and fit preprocessing only on training data; rerun baseline and new model.
3. Report revised results and uncertainty instead of selecting the best run.

The report distinguishes original and corrected settings, differing results and unresolved causes; rerunnability does not establish the claim.

Failure case and repair

Tune until test performance peaks, then claim generalization.

Test information enters selection and invalidates independent evaluation.

Freeze the plan, use an untouched evaluation or limit the conclusion.