Hi everyone, I’m trying to figure out the best way to build an AI assistant that can become a real expert on a complex software platform.
The goal isn’t just to build a chatbot that answers documentation questions. I want it to behave more like an experienced product expert. It should be able to answer questions, explain concepts, help users build workflows, diagnose errors, recommend best practices, explain configuration options, and generally guide users through the software.
The software has a few hundred pages of documentation covering features, automation actions, parameters, examples, troubleshooting guides, architecture, and best practices. There are also tutorial videos, knowledge base articles, and a large collection of existing automation scripts that are actively used. Ideally, the assistant should be able to leverage all of this knowledge to provide accurate, reliable, and well-grounded answers.
One important constraint is that I’d like the final solution to run locally, preferably on modest hardware without requiring a GPU for inference. Models under 3B parameter range seem realistic from a deployment perspective, but I’m not sure whether models that size can actually become experts on a complex software domain.
I’ve read about several possible approaches: Continued Pretraining (CPT) Supervised Fine-Tuning (SFT) RAG Agent-based systems with LangGraph
What I’m struggling with is understanding what the overall architecture should actually look like. If you’ve built something similar I’d really appreciate hearing about your experience or If you know of any similar projects, I’d love to check them out.
Some questions I’m wondering about: What architecture ended up working best? Can a relatively small local model realistically become an expert on a software platform, or is that expecting too much? Is RAG enough for this kind of problem, or would you still invest in CPT or fine-tuning? Would you use a single model, multiple specialized agents, or a different architecture altogether? If you were starting this project from scratch today, how would you design it? Thanks in advance !