Running an AI model on your own computer

A physical therapist in private practice spends her evenings writing up reports from scribbled notes. She'd like the AI to clean them up, but she doesn't want to send information about her patients to an online service. A colleague tells her: "Install a model at home, nothing leaves your computer." That's true, and it's doable. It's not as simple as it sounds, though, and the quality of the result isn't the same as with the big services.

A local model is a language model that you download and run on your own machine, with a dedicated program. Once installed, it works without a connection. Here is what it brings, what it costs in practice, and how to write suitable prompts.

What you gain

Your texts stay with you. What you type is processed by your computer and isn't sent to a provider. For sensitive notes, confidential drafts, or company documents, that's the main reason to go this route. One caveat: "local" doesn't mean "compliant." For health or customer data, legal obligations don't disappear because the processing happens on your own machine. If that's your case, talk to your data protection contact or a legal professional, and first look at what you shouldn't entrust to a chatbot.

You can work offline. On a train, on a construction site, in a building with no network, the model still responds.

You stay in control. The model doesn't change behavior overnight because of a provider update. Yours is frozen until you replace it, which makes a prompt more reproducible.

You can experiment without counting. No message cap set by a subscription. The limit is your machine's patience.

What you lose

Let's be clear: a model that fits on a desktop computer is much smaller than those running in the big services' data centers. In general, it reasons less well, follows long instructions less well, understands nuance less well, gets facts wrong more often, and handles very long texts less well. Depending on the machine, it can also be slow.

The tasks it handles well are rather simple and bounded: rephrasing a text, correcting, sorting, extracting information from a text you provide, summarizing a short document, making drafts. The tasks where it disappoints: multi-step reasoning, pointed general-knowledge questions, long, polished writing, and anything that has to be factually accurate without a source.

That's no small detail for anyone who relies on it without rereading. A small model makes things up more easily: you have to check every figure and every reference, as the article on reducing hallucinations explains.

What hardware you need

The deciding factor is memory. A model is a big file loaded into memory to respond, and if it doesn't fit, the machine becomes unusably slow. Three notions to know.

Size, in billions of parameters. You'll see models described as "3B," "7B," or "70B." The bigger the number, the more capable the model generally is, and the heavier.

Quantization. This is a compression that reduces the precision of the model's numbers so it fits in less memory, at the cost of a small loss of quality. Most models available for download come in several quantized versions, and the tool tells you the file size.

Available memory. On a computer, this is RAM, or the graphics card's memory if you have a dedicated one. On some recent Macs, memory is shared between the processor and the graphics part, which helps. A cautious rule: the model file must fit in your memory with a good margin, because room is also needed for the current conversation (the context window) and for the rest of your system.

With a recent computer and little memory, aim for small models. With a lot of memory or a good graphics card, you can go up in size. The tools often tell you whether a model has a chance of running on your machine: trust those warnings.

Tools for getting started

Two programs are widely used to begin with, and this guide doesn't try to tell you which is better.

• LM Studio has a graphical interface: you search for a model in a list, download it, and open a chat window. It's the simplest for a first try.
• Ollama works mainly from the command line and serves as a base for other applications. It suits you better if you're comfortable with a terminal or want to connect the model to other tools.

In both cases, the process is the same: install the program, choose a small model, download it (the file is measured in gigabytes, so allow for the space and a good connection), then chat.

A few precautions. Download models from the catalogs built into the tools or from recognized sources, not from a link you received in a message. Check the model's license if you plan to use it in a professional setting, because not all of them allow commercial use. And if you turn on features that browse the web or send data to a service, your processing is no longer entirely local.

Writing prompts for a small model

The principles remain those of writing a prompt, but a small model demands more discipline. A big service makes up for your approximations; a small one suffers from them.

One task per message. Rather than "summarize, correct, and translate," chain the three steps, as in a prompt chain.

Short, positive instructions. A list of twelve rules will be half forgotten. Keep three or four essential instructions.

An example. Small models gain a lot from seeing what's expected. The technique is explained in the article on few-shot prompting.

The text to process, supplied and delimited. Paste the material between clear markers, and ask it not to add anything.

A simple format. A bulleted list or a few lines. A highly structured format succeeds less often than with a large model.

For the physical therapist, here is a prompt that follows these principles. She replaces names with initials and keeps only what's strictly necessary:

You clean up physical therapy session notes.

Rules:
- Write three sentences maximum, in the present tense.
- Add no information that is absent from the notes.
- If a piece of information seems to be missing (duration, area treated), write [TO BE CONFIRMED].

Example.
Notes: R knee, pain 6/10 -> 4/10, mobilization, quad strengthening, see again Thursday
Report: Session focused on the right knee. Pain went from 6 to 4 out of 10. Mobilization and quadriceps strengthening; next session Thursday.

Notes to clean up:
[YOUR NOTES]

She reads each report before filing it: a small model can swap right and left or round a number. A writing aid doesn't replace the review by the professional who signs.

If the results are disappointing, try these in order: shorten the prompt, add a second example, move to a bigger model if your memory allows. Settings like temperature are explained in the article on parameters; they're often available in the local interface and useful for making responses more stable.

Local or online service: how to choose

Stay with an online service if you need the best reasoning quality, long documents, or web searches. Switch to local if the number one criterion is that the text doesn't leave your machine, if you work without a network, or if you want to learn how these models work. Many people mix the two: local for sensitive notes and simple tasks, the online service for the rest, with cleaned-up data. If you want to understand what happens inside, the article on how a language model works gives you the basics. And for a European service with no installation, there's also Mistral's Le Chat.

A trial tonight

Install LM Studio or Ollama, choose the smallest model recommended for your machine, and give it a task you know by heart: rephrasing a three-line email. If the result is correct, try a longer text. If your machine struggles, that's a sign you need a smaller model, not a longer prompt.