Skip to content
tishad.net
SearchCtrl K

Lessons / Part 7. Python, notebooks and the tools around them

39 A long run writes to disk as it goes

Each configuration lands in a results file the moment it finishes, so a dead kernel costs one case and the run resumes.

What people usually ask

Here are my 200 configurations for the spray dryer model. Run all of them in the notebook and tell me which one gives the lowest specific energy.

What comes back

One cell with a loop over all 200 cases that holds every result in a list in memory and loses all of it when the kernel dies at case 140, four hours in.

Ask this instead

I have configs.csv (200 rows, one per configuration, column config_id) and the run table runs.csv. Write a notebook that runs <model function> for every configuration, in this order:
1. Run one case first. Print its wall time and the estimated total for 200. Stop there and show me.
2. On "go": append each result to results.csv as soon as it completes, one row with config_id, inputs, outputs and time taken, header written once. Do not hold results in a list to save at the end.
3. Before the loop, read results.csv if it exists and skip every config_id already in it, so a restart after a crash carries on from the last written row.
4. Print a progress line every ten cases: done, left, time elapsed.
5. No time.sleep anywhere. No try/except that hides a failed case; write the error text in a status column and move on.
6. Report the best case only after the file holds 200 rows.

Attach: configs.csv and runs.csv, with the model function in a cell or a .py file beside them.

Why it works: Every finished case is already on disk, and the skip step means a restart continues from the last written row.

Drafted with Fable 5.1 from my notes; chosen and edited by me.