60 Five checks on a small data model

What people usually ask

Is my model good enough to use?

What comes back

“Yes, an R2 of 0.85 is good for process data”, from the single score it was given and no new fit.

Ask this instead

I have catalyst_runs.csv: 33 runs, inputs calcination_temp,
metal_loading, support_area, reduction_time, target activity.
The model is the random forest pipeline in model.ipynb, scored
R2 0.85 on one 80/20 split.
Write checks.ipynb on the same 33 rows, one section per check:
1. Leave one out: held out predictions in one column, R2 and
   RMSE on that column, replicates kept in the same fold.
2. Bootstrap: 1000 resamples, the 2.5th and 97.5th percentile
   of the score on the rows each resample left out.
3. Shuffled target: 200 permutations, p value with its floor.
4. Learning curve: score against training size from 10 rows up
   to 33 under 5 fold, never fewer rows than the folds need.
5. Ablation: each input removed in turn, paired difference with
   its spread.
End with one table: check, what it says about this model, what
it cannot say, and the number of fits it cost. Print the total
fits and the wall time.
Do not change the model between checks. Do not drop rows. Do not
say whether the model is good enough; the use it is for is not
in this prompt.

Attach: catalyst_runs.csv and model.ipynb.

Why it works: Each check is a procedure with its own limit and its cost written beside it, so the judgement stays with the person who knows what the model is for.