Skip to content
tishad.net
SearchCtrl K

Lessons / Part 10. Coding agents, commands and skills

73 Test a skill against a run without it

A reference problem with planted faults and blind runs with and without the skill show whether the skill adds anything.

What people usually ask

Does the sop-check skill work?

What comes back

“Yes, it is working well”, from the same session that wrote the skill and has never seen it fail.

Ask this instead

Before answering: write test/reference_sop.md, a short procedure with six planted faults (a volume in the wrong unit, a step out of order, a missing glove line, a reagent concentration with no unit, a duplicated step number, a torque with no tolerance) and one line that looks wrong but is correct: the 0.5 M NaOH dilution. Write test/answers.md listing each fault with its line number and the pass criterion: every fault found, the correct line left alone.
Then run the check four times: with the skill and without it, on <model A> and <model B>, each in a fresh session whose file list excludes test/answers.md. Score each run against the answers.
Give me the four scores in one table. For each run, show the file list it was given as evidence it could not read the answers. Change nothing in the skill until the table exists.

Attach: The reference problem, once it is written, kept apart from the answers file.

Why it works: a run without the skill is the baseline the skill has to beat, and the planted correct line catches a skill that flags everything.

Drafted with Fable 5.1 from my notes; chosen and edited by me.