Local Inference with Large Language Models A business owner running a confidential M&A advisory service moved LLM proofreading of client reports to locally run open-weights models rather than sending documents to hosted AI providers, because uploading the files would breach the NDAs his firm signs with clients. The author said the reports are typically due Friday evening with drafts ready between 2 and 3pm that afternoon, leaving too little time for manual proofreading passes, and that locally run models, while smaller and less capable than frontier models, are a surprisingly good match for the workload. He added that he would never accept a computer telling him "Sorry Jacques, I can't let you do that. Local Inference with Large Language Models To head off any criticism: not a word of this text was written by AI, all the mistakes are my own. If you should spot any I’d be very happy if you let me know. This article assume basic familiarity with the terminology around LLMs and the various moving parts, I’ve tried to explain as much as possible but in case something is unclear feel free to reach out and I’ll expand the article or clarify in an email. Why? Why make a set to run an LLM locally if the bleeding edge is affordable and easily accessible? I’ve been playing with LLMs since the first open models were released and even though at the time the results weren’t all that impressive it was clear that it wasn’t going to stop there. Over the last couple of years the model capabilities have improved at a breakneck pace and today these models are extremely capable when used appropriately. The business that I run sells a service to investors and companies that are looking to be acquired that is highly confidential in nature. Effectively we ask other companies to share pretty much everything, both the good and the bad with our team so that we can come to a balanced verdict in our report. To enable this sharing we sign NDAs non-disclosure-agreements that usually have scary numbers in them in case we would breach that confidentiality, and rightly so, this kind of data is easily weaponized. As part of our process we write reports, our ‘work product’ and these reports need to be proofread. The problem is that typically these reports are due on Friday evening and our draft version is usually only ready somewhere between 2 and 3pm on Friday afternoon. That leaves very little time for proofreading, usually we try to get one or two passes in before the document is handed to a customer but it isn’t rare at all that on second or third reading during the weekend we still find mistakes. These can be small and innocent or the could be larger cf: typos and grammar errors or errors in the content itself . Given infinite time I’m sure we would find all of these but we simply don’t have that luxury. Now, the easy way out would be to upload the document just prior to sending it out to an AI service provider, have that service do the proofreading and then we would apply the corrections there is absolutely no way that I would allow an AI control or direct input into our work process . But that would cause us to break our NDA. And that’s before we get into the hairy details of questions such as ‘Can we trust Altman or Amodei when they say they won’t use the data for their training?’, ‘Even if we can trust them, can we trust everybody who works there?’ and ‘And even if we can, are we sure that shipping these files out at all is a wise decision?’. The answer the last one was the easiest, that was simply ‘No.’. So we decided to bite the bullet and do this locally. We have some other reasons as well: we don’t particularly like the idea of a large entity vacuuming up the whole of human culture and then selling it back to us, no matter how nicely packaged and how useful it is. That feels like a very unethical thing to do and the providers of ‘open weights’ models are closer to how we think this should be handled. Second to that is that we don’t want to end up in a situation where an outside party has control over our workflow and can tell us what we are - or are not - allowed to do. ‘Sorry Jacques, I can’t let you do that’ is not something that I would ever accept from a computer, computers do what I tell them to do, not what they allow. These locally run models are not precisely as good as the big frontier models. They tend to be a bit smaller, sometimes a bit older and probably not quite as clever. But depending on your workload and other demands such as confidentiality they can be a surprisingly good match. And in case the likes of OpenAI decide to hack your server they may well be your only means of defense https://codely.com/en/cafe/hack-hf-va72 . What do you need to do local inference? If you want to do inference locally you’re going to need a couple of things: - A piece of hardware powerful enough to do this on - A model that will fit that hardware - An inference engine that will push your input through the model to generate the required output Let’s start with the last one. Inference Engines There are a small number of major engines suitable for deployment on your own hardware, a much larger number of derivatives sometimes weakly derived, sometimes not much more than a relabeling and then there are numerous smaller efforts. We only considered open source engines, and more specifically engines that we have actually used benchmarked, and used in production . Let’s focus on the main engines first. These are the ones that we used that are either the ‘root’ projects or independent projects that have shown sufficient staying power that you could presumably build a business on them: Llama.cpp Llama.cpp https://github.com/ggml-org/llama.cpp/ WP https://en.wikipedia.org/wiki/Llama.cpp is probably one of the easiest ways to get started with local llms. The project is now 3