Lessons / Part 6. Running a model on your own computer
31 A local model reads what you give it, at the context you set
Set the context length and GPU offload before pasting a long document to a local model, then keep the answer inside the pasted text.
What people usually ask
Summarize this 80 page report and check online whether the limit in
section 6 is still current.
<report pasted>
What comes back
“Too long”, or a summary of the first pages, or a summary of pages the model never saw; the online check comes back as a guess, because a local model cannot browse.
Ask this instead
First, in LM Studio, before the chat, in the model's load settings:
context length: enough for the text, about 1.3 tokens per word
(80 pages at 400 words is 32000 words, so set 48000 or more).
GPU offload: max. On a 24 GB card pick a 30B class model.
The same from a terminal:
lms load <model> --context-length 48000 --gpu max
Then:
Below is the full text of <report name>, pasted.
Summarize it in 300 words, sections in the order they appear.
Use only the pasted text. Where a figure or date is not in it,
write "not in the text". Do not look anything up; you cannot
browse, and every claim will be checked against the pages.
After each paragraph, the page numbers it came from.
<the report as plain text>
Attach: The document as plain text, pasted into the chat, not as a file.
Why it works: A local model holds only what fits in the context set at load time and knows nothing past its training, so the settings give it room for the whole text and the prompt keeps it inside that text.
Sources
- https://lmstudio.ai/docs/app/advanced/per-model
- https://lmstudio.ai/docs/cli/load
Drafted with Fable 5.1 from my notes; chosen and edited by me.