Ok to start with I am not a programmer or an IT person whatsoever. I have spent the last few months learning as much as I can about AI. I have been using Coursera to study topics on the various LLMS, CISSP, Python etc and continue to do so. I have a Claude Pro account that I am using to help me set this LLM up and have a temporary Gemini pro account via Coursera. I am trying to get a good general background on AI and security but I am still ignorant in so many ways.
Hardware:
Macbook M5 Max, 18 core cpu, 40 core gpu, 128GB unified memory, 4TB hd, OS Tahoe
Will eventually use Tailscale for remote access
Stack:
Docker Desktop using Docker Model Runner for local inference (full Metal GPU/unified memory access currently). Open webUI as chat interface, in Docker via Compose
Models:
Gemma 4 (~12B) - daily use
Qwen3 30B-A3B-Q4_k_m - deep research
RAG:
SentenceTransformers embedding, default function calling mode. Multiple topic-based knowledge collections (RV/camping, photography gear, truck, home equipment, etc) each one of these contain AI written md files with full manufacturer manuals pdf.
Other tools:
DrawThings - image/video generation
MacWhisper Pro - audio/video transcription
Kokoro TTS - local voice output
What is my objective?
So far this whole setup has been guided by Claude and has been a painful process but a learning one. I most likely went about this the wrong way and i’m ok with that but I feel like my next step is to make sure my models work the way I need them to and start using them more than Claude. I’m just not sure at my skill level where to go and what to do without it being too technical. I feel like there is a normal path that people follow but I don’t know what it is.
I will likely be asking lots of questions of you guys, thank you,
Tim