Lessons / Part 4. Documents and data with an assistant
26 Catalogue a large archive by script before a model reads it
A script lists and ranks thousands of files by metadata and term counts, and a model reads only the slice that ranks high.
What people usually ask
Read these 4000 reports and find the ones about corrosion failures.
What comes back
A short list of matches from the few files that were actually opened, with a sentence claiming all 4000 were read.
Ask this instead
Do not read any report. Write a Python script, catalogue.py, that walks <path to archive> without changing anything and writes catalogue.csv with one row per file: path, file name, extension, size in bytes, modified date, and for PDF and Word files the title and author from the file properties, blank where absent.
Then write rank.py: for each file, count the whole word occurrences of every term in terms.txt (corrosion, pitting, cracking, leak, failure) in the file's text, and write the counts and a total to ranked.csv.
Print the top 30 by total with their counts and file names. No model reads anything in this step.
Stop there. In the next session I will pick one slice of the top 30, and you will read only the pages of those files where the terms occur, not the whole files.
Attach: The folder path and terms.txt; nothing opened yet.
Why it works: A script treats all 4000 files the same way and leaves a CSV that shows what was counted, so the model’s reading later starts from a ranked list and covers a slice it can actually hold.
Drafted with Fable 5.1 from my notes; chosen and edited by me.