Skip to content
tishad.net
SearchCtrl K

Lessons / Part 8. Machine learning on small data in plain terms

56 Before a second dataset can test a model

Every column in both tables is matched on what it measures and on what basis before any score is run, and a poor score is reported as poor.

What people usually ask

Test my model on this second dataset.

What comes back

A low score, then a paragraph on how the second vessel is different, written so the model survives.

Ask this instead

Two tables. train_runs.csv is the 40 runs the model in
model.ipynb was fitted on, from a 5 L stirred vessel.
check_runs.csv is 18 runs from a 50 L vessel in another lab.
Target is conversion_pct in both.
Before any scoring, build one table with a row for every column
in either file: the column name in each file, what was measured,
on what basis (wet or dry, mass or volume, per batch or per
hour), the unit, the min and max in each file, and any constant
folded into the number (a density, a calibration factor). Write
"not stated" where the attached descriptions do not say.
Then list every row of check_runs.csv that falls outside the
training min and max on any input, with the column and value.
Stop and show both tables. Score nothing until the columns are
confirmed to mean the same thing.
When the score is run and it is poor, report it as poor. No
paragraph on why the vessels differ unless it names a column
mismatch from the first table.

Attach: Both CSV files and a short description of each column in each file.

Why it works: The test is only a test once the two tables are known to measure the same thing on the same basis, and a bad result is allowed to stand.

Drafted with Fable 5.1 from my notes; chosen and edited by me.