Skip to content
tishad.net
SearchCtrl K

Lessons / Part 4. Documents and data with an assistant

26 Catalogue a large archive by script before a model reads it

A script lists and ranks thousands of files by metadata and term counts, and a model reads only the slice that ranks high.

What people usually ask

Read these 4000 reports and find the ones about corrosion failures.

What comes back

A short list of matches from the few files that were actually opened, with a sentence claiming all 4000 were read.

Ask this instead

Do not read any report. Write a Python script, catalogue.py, that walks <path to archive> without changing anything and writes catalogue.csv with one row per file: path, file name, extension, size in bytes, modified date, and for PDF and Word files the title and author from the file properties, blank where absent.
Then write rank.py: for each file, count the whole word occurrences of every term in terms.txt (corrosion, pitting, cracking, leak, failure) in the file's text, and write the counts and a total to ranked.csv.
Print the top 30 by total with their counts and file names. No model reads anything in this step.
Stop there. In the next session I will pick one slice of the top 30, and you will read only the pages of those files where the terms occur, not the whole files.

Attach: The folder path and terms.txt; nothing opened yet.

Why it works: A script treats all 4000 files the same way and leaves a CSV that shows what was counted, so the model’s reading later starts from a ranked list and covers a slice it can actually hold.

Drafted with Fable 5.1 from my notes; chosen and edited by me.