Lessons / Part 8. Machine learning on small data in plain terms
47 Tree ensembles in plain terms
Three tree models under identical folds, with the spread and a probe past the data, instead of a winner from one split.
What people usually ask
Random forest or XGBoost, which is better for my data?
What comes back
XGBoost declared the winner on one train test split, with a score that changes when the split does.
Ask this instead
I have carbon.csv: 42 batch adsorption runs, columns pH, c0_mg_L, time_min and q_mg_g (uptake). Fit three models to q_mg_g with scikit-learn defaults: DecisionTreeRegressor, RandomForestRegressor, GradientBoostingRegressor.
Use the same RepeatedKFold(n_splits=5, n_repeats=10, random_state=0) for all three. Report one table: model, RMSE mean, RMSE standard deviation across the 50 folds.
Then build five rows where c0_mg_L is 1.2 times the largest value in the table and the other inputs sit at their medians. Print each model's prediction for those rows beside the largest q_mg_g ever measured, so the flattening past the data can be seen.
Write one paragraph per model in plain words on what the table shows for it. No claim that the table does not support.
Do not drop rows. Do not install XGBoost; gradient boosting from scikit-learn is enough here. If two models sit inside each other's spread, say they are not separated.
Attach: The run table as CSV.
Why it works: Identical folds and the spread show whether the gap between models is larger than the noise, and the rows at 1.2 times the largest input show what a tree does where it has no data.
Drafted with Fable 5.1 from my notes; chosen and edited by me.