We’ve realized that we use Claude and OpenAI models as capable web search and summarization engines with a bit of actual coding and system administrator work sprinkled in. While the offerings has been great, we are not that happy to share what is becoming quite a comprehensive amount of data about us and our lives over to the companies behind these models. Therefore we would like to run a local LLM/AI setup at home for our selves.
I have been experimenting with different models, backends and harnesses, and these have been working good. The issue however is the limitations with regards to context and model size when running on a single 24GB 7900xtx card and a system with 64GB ddr5 memory. The same system which I would like to do other stuff on as well and I am not that keen on having the fans going off all the time either.
Originally I considered getting a Strix Halo 128GB system from minisforum, but I do see some mixed reviews on these systems, which in turn makes me look at the other offerings. I am hoping someone here has some experience or insights that they want to share regarding what I am currently considering getting.
minisforum ms-s1 max (4600 USD) / ASUS Ascent GX10 (6100 USD) / New mac with 128GB (7100 USD).
If you were going to get a simple, two user setup capable of running decent models that would work with in an overall setup that supports memory, web search and some work related operations, which would you go for and why?**