I recently started setting up a self-hosted LLM on my home server. I’m running Mistral 7B because my home server machine was new when Infinity War was in theaters.
I’m getting my RAG stuff set up, and something which I need help with is Linux support. I don’t mean “what’s the thing I type to list the stuff in the folder?” I mean “I have a power management issue causing a UCSI error, help me diagnose the problem and resolve.” So I’m looking at starting a RAG database of documents for my distro so I can more securely and privately copy and paste command line output in there without worrying about who knows the name of my user, machine, MAC address, etc.
With both the recommendations of people coming over to Linux from Windows, and then wide variety of Linux distros recommended, and the relative ease of setting up Ollama and a model, this seems like an easy stepping stone of support. Or, maybe I overestimate people’s general technical competency. Maybe both.
Curious of thoughts on the subject, and maybe this is even something that’s already out there and I’ve missed it.