Stress-test a research design with AI

Choose the inferential target before the model

Predicting who joins a research programme differs from estimating whether it improves ability. Prediction needs appropriate generalization checks; causal estimation needs a defensible counterfactual comparison. Qualitative inquiry may examine how experiences are understood. Specify the target before asking AI for candidate methods and assumptions.

Ask AI for failure cases, not just code

Give AI a draft and ask for concrete processes that would make its conclusion misleading: baseline motivation differences, the same student in train and test, or adjustment for a post-treatment variable. Decide whether those mechanisms are plausible. Diagrams and simulations must state variable meanings, timing, true parameters and expected behaviour.

Check simulations before interpreting results

Simulations provide known settings. Test zero effect, known effect and an assumption violation, checking expected estimator behaviour. Successful execution does not validate design, units or statistical units. Hand-check a tiny case and distinguish training randomness, sampling uncertainty and model uncertainty.

Decide what could change the conclusion

Before main results, record outcomes, exclusions, comparison budgets, stopping rules and sensitivity checks. Justify sample size against information goals. Result-informed exploration can be useful, but retain revisions rather than presenting them as prespecified. Another person should be able to reconstruct the main result from materials and configuration.

Worked example · fictional

From a problem to inspectable work

Fictional example: a model predicts grades accurately, so a team plans to claim an AI study tool improves grades.

1. A predictor may use prior ability and motivation; accuracy does not identify an intervention effect.
2. For prediction, split according to deployment and prevent student leakage. For causality, define intervention, comparator, time zero and identification assumptions.
3. Simulate motivation affecting tool use and grades with zero true tool effect; inspect whether a naive comparison still differs.

Design memo: define a causal target, but current observations cannot rule out unmeasured motivation. Report association and sensitivity. Keep the zero-effect simulation as a bias demonstration, not real-world causal evidence. Next inspect allocation or better baseline information.

Failure case and repair

Choose a complex model before defining the question.

Analysis may not identify the effect or explain the phenomenon.

Define the target, population, comparison and assumptions first.