59 An ablation study in plain terms

What people usually ask

Does adding the pH column help?

What comes back

“Yes, the score improved from 0.76 to 0.79”, one fit each, with no way to tell whether 0.03 is anything.

Ask this instead

I have fermentation_runs.csv: 29 runs, inputs temp_C, feed_rate,
dissolved_O2, pH, target titre_gL. The model is the pipeline in
model.ipynb.
Run an ablation on pH. Fit the full model and the model without
pH under identical 5 fold cross validation: same folds, same
seed, same settings. Repeat the fold split with 20 different
seeds for both.
Report the mean cross validated RMSE for each model, and the
mean of the paired differences across the 20 seeds with its
standard deviation. Plot the 20 paired differences as points on
one axis with zero marked.
If the mean difference is smaller than its standard deviation,
write "no finding at 29 runs" and nothing more about pH.
If pH does not help, keep two statements apart: "pH is not
needed given temp_C, feed_rate and dissolved_O2" is what this
test can say; "pH has no effect on titre" is not. Print the
correlation of pH with the other inputs beside the result.
Do not retune the reduced model.

Attach: fermentation_runs.csv and model.ipynb.

Why it works: The comparison is paired, repeated and reported with its spread, and the sentence the test cannot support is named before anyone writes it.