Lessons / Part 8. Machine learning on small data in plain terms
45 Flag outliers, do not remove them
Flagged rows come back with their run ids for a look in the lab notebook, and a run leaves the table only with a recorded fault.
What people usually ask
Remove the outliers from my data before modelling. Use the usual 3 sigma rule.
What comes back
A table of 14 rows where there were 18, a better score, and no record of which runs went or why.
Ask this instead
runs.csv has 18 rows, one per run, with run_id. Flag, do not remove.
1. For each numeric column, compute the modified Z score (0.6745 times the distance from the median, divided by the median absolute deviation) and mark values beyond 3.5. Also mark values outside the box plot fence (1.5 times the interquartile range beyond either quartile).
2. Print every flagged row: run_id, the column, the value, the column median, and which rule fired.
3. Print the largest plain Z score any one value can reach in a table of 18 rows, (n minus 1) over the square root of n, beside the 3 that the sigma rule uses.
4. Write the flags to a new column outlier_flag in runs_flagged.csv. runs.csv stays as it is.
5. Remove nothing. Each flagged run will be looked up in the lab notebook, and a run is dropped only when a recorded fault explains it, with the reason written in a notes column.
Attach: The run table as runs.csv.
Why it works: A flag with a run id can be checked against the notebook, and the ceiling on the plain Z score shows how little room a 3 sigma rule has in a small table.
Drafted with Fable 5.1 from my notes; chosen and edited by me.