54 Shuffle the target and refit

What people usually ask

Is my R2 of 0.8 real?

What comes back

“Yes, 0.8 is a strong fit for experimental data”, with nothing run to check it.

Ask this instead

I have coating_runs.csv: 26 runs, inputs line_speed, oven_temp,
viscosity, solids_pct, target thickness_um. The gradient boosting
pipeline in model.ipynb scores R2 0.80 under 5 fold cross
validation.
Run a permutation test on that score. Shuffle the thickness_um
column 200 times. For each shuffle, refit the same pipeline under
the same 5 folds (same fold seed) and record the cross validated
R2. Change nothing else.
Plot the 200 shuffled scores as a histogram with the real 0.80
marked by a vertical line. Print how many shuffled scores reached
0.80 or above, and the p value as (that count + 1) / 201. Say in
one line that 200 shuffles cannot give a p value below 1/201.
Then one sentence on what this test rules out (a score this high
by chance on 26 rows) and one on what it does not (whether the
model is accurate enough for its use).
Use the real rows only. Do not shuffle the inputs, and do not
tune anything between shuffles.

Attach: coating_runs.csv and model.ipynb.

Why it works: The question “is it real” becomes a count of how often chance beats the score, and the floor of that count is printed with it.