{"slug": "going-squirrel-on-this-new-ish-hobby", "title": "Going squirrel on this new.....ish hobby", "summary": "A hobbyist returning to homelabbing after 10-15 years built a Proxmox-based local AI setup running llama.cpp with a Hermes agent on an Ubuntu Server VM, reporting the system operational in a November 8, 2026 update. The build pairs an Intel Core5 Ultra 225f with 32GB DDR5 and a 16GB Nvidia GeForce RTX 4060Ti, and runs Qwen3.8-27B-GSQ-RCO-IQ3_S.gguf at a 131072 context size with 99 GPU layers, q4_0 KV cache and flash attention enabled. The user said they were unhappy with DeepHermes, MedGemma and Gemma4 and plans to swap models, while keeping the AI system walled off from personal files unless moved manually.", "body_md": "AI Disclosure:I will be using AI to search for ideas and suggestions on ways to fix something I break. I unfortunately have only so much time and see it as a tool like you would a calculator.\n\nWhere do I “project start”…… I use to be into this stuff 10-15 years ago, but family, work, and volunteer FF took over. Mental health tanked a couple years ago, finally getting out of my funk and not just watching TV to pass the time.\n\nGoal/project/pipe dream\n\n1. Use proxmox and setup a secure host with backups to NAS.\n2. Use Ubuntu server  and setup a llama.cpp server with Hermes agent for a 100% local ai digital assistant. Multiple models for different purposes.\n3. Setup homarr on my NAS and make a nice dashboard.\n4. Program an esp32 to display proxmox system info.\n5. I don’t know…… first Devember……\n\n \n4 Likes\n\n \nClarification - Ubuntu will be a VM in proxmox.\n\n \n\n \nWelcome back. Since Ubuntu is going to be a VM anyway, figure out early how the NAS storage gets into it, because sharing a drive across VMs and LXCs in Proxmox took special config for me where plain Ubuntu just mounts it. That’s a big part of why I ended up on plain Ubuntu Server with SSH for spinning things up and tearing them down, and I’d only reach for Proxmox again for a bigger deployment. If you stay with Proxmox, get the backups to the NAS working and restore one VM before you start on llama.cpp, so a broken experiment is cheap to undo.\n\n \n1 Like\n\n \nMy thoughts are I want the Ai system not to be able to access personal files unless I move them myself. I just want to backup the system for when I break something. I feel it’s easier to restore a backup then rebuild something……\n\nI guess I am gun shy one losing files…… I know I am also bad for no off-site backup.\n\n \n1 Like\n\n \nOK, I got off work early and wanted to show what I am working with. I heard a janitor likes pictures……\n\n### \n\nCore5 ultra 225f\n\n32 GB RAM  ddr5\n\n16GB NVIDIA GeForce RTX 4060Ti\n\nAsus B860m max gaming ax motherboard\n\n1tb nvme\n\n512mb nvme\n\nModded Rosewill RSV-R4100U server chassis ([https://makerworld.com/en/models/1262238-rosewill-rsv-r4100u-replacement-front-panel?from=search#profileId-1287123](https://makerworld.com/en/models/1262238-rosewill-rsv-r4100u-replacement-front-panel?from=search#profileId-1287123))\n\nI bought a pre built PC from Costco because I could not beat the price on clearance. It had a intel B570 that i swapped for the 4070. Took everything out of the case it came in and put it in the modified case.\n\n### \n\n4x WD RED 6TB - in two RAID1 for storage and backup (1-Data and pictures 2-ISO’s and Proxmox backups)\n\n2 x Patriot P300 256gb - in RAID1 for VM’s and docker\n\n16GB RAM\n\nRack mounted ([https://makerworld.com/en/models/2346242-ugreen-dpx4800-plus-nas-19-inches-rack-mount?from=search#profileId-2565315](https://makerworld.com/en/models/2346242-ugreen-dpx4800-plus-nas-19-inches-rack-mount?from=search#profileId-2565315))\n\nN97 mini PC - WANker\n\nUsing as a device that is exposed to the World via Cloudflare, hosing a personal website with wordpress.\n\nAnd a CyberPower CP2000PFCRM2U PFC Sinewave UPS in a Startech rack.\n\nI hope to have the Ai PC up and running with Hermes by then end of the weeked. If you have any questions or comments fell free.\n\n \n\n \n**Nov 8th 2026 update** - I have the Ai PC up and running. Slowly getting more comfortable in Proxmox. Used a lot of help from google to get the GPU to work. Had to play around with settings to get the Qwen 3.8 to run with a large context for hermes. I am going to change the models around as I am not happy with DeepHermes, MedGemma, or Gemma4. This is what I have currently in my **model.ini** file.\n\n*[*]\nn-gpu-layers = 99\nbatch-size = 256\nubatch-size = 256\nflash-attn = on\ncache-type-k = q4_0\ncache-type-v = q4_0\njinja = on*\n\n*[qwen27b]\nmodel = /home/XXXXX/llama.cpp/models/Qwen3.8-27B-GSQ-RCO-IQ3_S.gguf\nn-gpu-layers = 99\nctx-size = 131072\nbatch-size = 128\nubatch-size = 128\ncache-type-k = q4_0\ncache-type-v = q4_0\nflash-attn = on*\n\n*[deephermes8b]\nmodel = /home/XXXXX/llama.cpp/models/DeepHermes-3-Llama-3-8B-q4.gguf\nn-gpu-layers = 99\nctx-size = 262144*\n\n*[medgemma4b]\nmodel = /home/XXXXX/llama.cpp/models/medgemma-1.5-4b-it-UD-Q8_K_XL.gguf\nn-gpu-layers = 99\nctx-size = 262144*\n\n*[gemma4]\nmodel = /home/XXXXX/llama.cpp/models/gemma-4-12b-it-Q4_0.gguf\nn-gpu-layers = 99\nctx-size = 262144*\n\nAnd this is my **llama-server.service**\n\n*[Unit]\nDescription=Llama.cpp Server Instance\nAfter=network.target local-fs.target nvidia-persistenced.service multi-user.target\nWants=nvidia-persistenced.service*\n\n*[Service]\nType=simple\nUser=XXXXX\nGroup=XXXXX\nWorkingDirectory=/home/XXXXX/llama.cpp\nExecStartPre=/bin/sh -c ‘nvidia-smi >/dev/null 2>&1 || exit 1’\nExecStart=/home/XXXXX/llama.cpp/build/bin/llama-server\n–models-preset /home/XXXXX/llama.cpp/models/models.ini\n–models-max 1\n–port 8080\nRestart=always\nRestartSec=15*\n\n*[Install]\nWantedBy=multi-user.target*\n\nHermes is also operational with telegram and web search integration. I plan on researching skills this week at work to try that next weekend, and try a couple different models.\n\n \n1 Like", "url": "https://wpnews.pro/news/going-squirrel-on-this-new-ish-hobby", "canonical_source": "https://forum.level1techs.com/t/going-squirrel-on-this-new-ish-hobby/260063#post_7", "published_at": "2026-11-16 12:10:43+00:00", "updated_at": "2026-11-16 12:23:03.404890+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-tools", "ai-infrastructure"], "entities": ["Proxmox", "llama.cpp", "Hermes", "Ubuntu Server", "Qwen3.8-27B-GSQ-RCO-IQ3_S.gguf", "Nvidia GeForce RTX 4060Ti", "Intel Core5 Ultra 225f", "DeepHermes"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/going-squirrel-on-this-new-ish-hobby", "markdown": "https://wpnews.pro/news/going-squirrel-on-this-new-ish-hobby.md", "text": "https://wpnews.pro/news/going-squirrel-on-this-new-ish-hobby.txt", "jsonld": "https://wpnews.pro/news/going-squirrel-on-this-new-ish-hobby.jsonld"}}