Are Local LLMs Really That Good?

This is a question that many AI users are probably asking. The reason is simple: why should I pay for something if I can run it myself? But the answer is not that simple.

There are plenty of sources around the internet, so rather than giving you a concrete decision, I want to present one principle:

Run the numbers and do the math for yourself

No one is in exactly your situation. You need to know what you are buying in terms of hardware for your local models, how you want to use it, and what you are sacrificing by not using OpenAI or Anthropic services.

Because let's be real: expecting to buy $4,000 worth of hardware to run a Gemma or Qwen model and get the same level of "intelligence" as Fable or Astra is delusional.

I am not saying that you cannot do it with a big budget. For some companies, it may be worth investing over $100,000 to build an AI solution that runs Kimi 3.0. For others, one DGX Spark may be enough instead of paying $10,000 in API costs.

Just as a company would run the numbers and make decisions based on them, you should too. Do not make your decision solely out of fears such as:

We could lose access to the APIs of the best models because of...
Companies are getting our data and learning from it.
Data centers need a lot of electricity and water for cooling.

I am not saying that those are not relevant points, but if you are using a $20 subscription for your side project, it does not make sense to pursue local AI solely because of the price.

Real example

I am considering buying a new Mac Mini that is coming out in a couple of weeks: a 64 GB M5 Pro version, on 22 September. But with everything else I have going on, I am not even maxing out a $20 subscription. Even if I were to max out a $100 subscription, it would take three years to match the price of the Mac Mini.

Does that mean I should not buy it? It depends on what I want to use it for and what my fears are.

I am based in Europe. Do I have other options if the US decides to cut the rest of the world off from its AI?
Am I doing research involving sensitive data that should not fall into the wrong hands?
Am I willing to accept less intelligence in exchange for having it in my own hands and with a lower environmental impact?
Do I want to learn how to manage local AI and experiment with it?
Do I fear that I will not be able to buy it later, or that it may cost double the price?

What I want you to take from this article is this: think about why you want local AI and what your expectations are. Then do your research, because you are probably not the only one thinking the same way, and run the numbers.

See you next Monday.