Skip to content
tishad.net
SearchCtrl K

Lessons / Part 2. Context and memory

12 Cut a big job into steps that each leave a file

A 300 sheet conversion done in batches of 20, each saved to disk, so a stop costs one batch and the next prompt resumes from the log.

What people usually ask

Convert all 300 scanned run sheets into one table.

What comes back

Twenty sheets done and then a stop with nothing saved, or a usage limit reached part way with the work gone.

Ask this instead

The folder scans/ holds 300 scanned run sheets named sheet_001.png to sheet_300.png. Work in batches of 20 in file name order.
For each batch write batch_NN.csv with the columns run_id, date, operator, feed_kg, temp_c, yield_pct, and a flags column for any cell you could not read with confidence. Then append one line to progress.log: batch number, row count, flagged cell count.
Do one batch, then stop and report the rows written and the cells flagged. Do not start the next batch.
When I say "next", read progress.log first and do the lowest batch number not listed there. Never redo a batch that is listed.

Attach: The folder of scans and progress.log, empty at the start.

Why it works: Each batch leaves a file and a log line, so whatever stops the run, the next prompt starts from the record of what is done.

Drafted with Fable 5.1 from my notes; chosen and edited by me.