learning Generative AI/LLM through practical projects

A developer outlines a practical project roadmap for learning generative AI and large language models, progressing from basic LLM usage to retrieval-augmented generation (RAG) and fine-tuning with LoRA/QLoRA. The approach emphasizes hands-on projects, such as building a lead qualification system and a document Q&A bot, and references resources like LangChain's RAG from Scratch and fine-tuning repositories.

If you want to become good at Generative AI/LLM engineering , watching tutorials is not enough. The fastest way to understand these technologies is to build projects where you are forced to solve real problems: prompting, context management, retrieval, model adaptation, and evaluation. Here is a practical project roadmap: LLM → RAG → Fine-Tuning → Evals Build an AI system that receives a new customer lead and decides: Input: "Hi, I'm looking for an enterprise plan for 200 employees. We need SSO and would like to schedule a demo next week." Output: Lead Quality: High Intent: Enterprise Purchase Company Size: 200 employees Urgency: High Recommended Action: Schedule Demo This project gives you a strong foundation in: Before jumping into RAG or fine-tuning, you should understand how an LLM behaves without external knowledge or model customization . This gives you the baseline against which you can later compare RAG and fine-tuning. Now take the same LLM and give it access to your own knowledge base. Upload documents such as: Company Policies Product Documentation HR Policies FAQs Technical Documentation Pricing Documents Users should be able to ask questions about these documents. User: "What is our work-from-home policy?" RAG pipeline: User Question ↓ Query Embedding ↓ Vector Database ↓ Retrieve Relevant Documents ↓ Context + Question ↓ LLM ↓ Grounded Answer RAG works by retrieving relevant information from an external data source and providing that information to the LLM as context. GitHub https://github.com/langchain-ai/rag-from-scratch?utm source=chatgpt.com Build the project in stages: Level 1 — Basic RAG Level 2 — Better RAG Level 3 — Production RAG A great reference is LangChain — RAG From Scratch https://github.com/langchain-ai/rag-from-scratch?utm source=chatgpt.com . It builds RAG progressively from indexing, retrieval and generation, making it particularly useful for understanding how RAG actually works rather than simply copying a framework implementation . GitHub https://github.com/langchain-ai/rag-from-scratch?utm source=chatgpt.com You can also explore LlamaIndex RAG example https://github.com/danielbank/rag-llamaindex?utm source=chatgpt.com for a more application-oriented implementation. GitHub https://github.com/danielbank/rag-llamaindex?utm source=chatgpt.com Important:This should be treated as an educational AI project, not a real medical diagnostic system. The goal is to take an open-source LLM and adapt it to produce responses in a particular domain and format. For example, create a dataset containing: Question ↓ Medical Context ↓ Expected Response Then fine-tune an open model on your dataset. Input: "What are common symptoms associated with iron deficiency?" The model should learn to produce a response following your desired structure and style. This project teaches: Instead of trying to fine-tune a huge model from scratch, start with LoRA/QLoRA . These techniques make experimentation much more practical. For a simple introduction: Fine-Tuning LLMs with LoRA and QLoRA https://github.com/Apoorva-Udupa/Fine-Tuning LLMs with LoRA and QLoRA?utm source=chatgpt.com This repository demonstrates LoRA and QLoRA fine-tuning using PyTorch and Hugging Face Transformers. GitHub https://github.com/Apoorva-Udupa/Fine-Tuning LLMs with LoRA and QLoRA?utm source=chatgpt.com For a more complete implementation: LLM Fine-Tuning — SFT, LoRA & QLoRA https://github.com/gazelle93/llm-fine-tuning-sft-lora-qlora?utm source=chatgpt.com It includes dataset loading, tokenization, SFT, LoRA and QLoRA examples. GitHub https://github.com/gazelle93/llm-fine-tuning-sft-lora-qlora?utm source=chatgpt.com Don't think: Fine-tuning = giving the model more knowledge Instead, think: Fine-tuning = adapting model behavior, style, format or task performance. For frequently changing factual knowledge, RAG is often a better solution. This is the project most beginners skip. And it is one of the most important. Suppose your RAG system answers: "What is the company's leave policy?" How do you know whether the answer is actually good? You need an evaluation system. Create a test dataset: Question Expected Answer Retrieved Context Generated Answer Then evaluate the system automatically. Retrieval Generation Question: What is our annual leave policy? Expected: Employees receive 24 days of annual leave. Model Answer: Employees receive 24 days of annual leave. Evaluation: Correctness: 1.0 Faithfulness: 1.0 Relevance: 1.0 Now intentionally introduce a bad answer: Model Answer: Employees receive 30 days of annual leave. Evaluation: Correctness: 0.0 Faithfulness: 0.0 You have now started building an LLM evaluation pipeline . A good reference is RAG Evaluation Framework https://github.com/Aftabbs/RAG-Evaluation-Framework?utm source=chatgpt.com . It separates evaluation into retrieval quality and generation quality and uses LLM-based evaluation with LangChain. GitHub https://github.com/Aftabbs/RAG-Evaluation-Framework?utm source=chatgpt.com Another useful project is Ragas https://github.com/vibrantlabsai/ragas?utm source=chatgpt.com , which provides metrics and test-data generation for evaluating LLM applications and RAG systems. GitHub https://github.com/vibrantlabsai/ragas?utm source=chatgpt.com You can also study LLM RAG Eval https://github.com/sujitpal/llm-rag-eval?utm source=chatgpt.com , which focuses specifically on evaluating RAG pipelines. GitHub https://github.com/sujitpal/llm-rag-eval?utm source=chatgpt.com Instead of building four unrelated projects, build them as a progression: GENERATIVE AI │ ▼ ┌─────────────────────┐ │ 1. Lead Triaging Bot│ │ LLM │ └──────────┬──────────┘ │ ▼ ┌─────────────────────┐ │ 2. Knowledge │ │ Assistant │ │ RAG │ └──────────┬──────────┘ │ ▼ ┌─────────────────────┐ │ 3. Medical Advisor │ │ Fine-Tuning │ └──────────┬──────────┘ │ ▼ ┌─────────────────────┐ │ 4. Evaluation │ │ Framework │ │ Evals │ └─────────────────────┘ | Project | Technology | You Learn | |---|---|---| Lead Triaging Bot | LLM | Prompting, structured output, tools | Knowledge Assistant | RAG | Embeddings, retrieval, vector DB, RAG | Medical Advisor | Fine-Tuning | SFT, LoRA, QLoRA, datasets | Evaluation Framework | Evals | Metrics, test datasets, hallucination detection | But the real value comes when you connect them . Your final architecture can look like: User │ ▼ Lead / Query │ ▼ ┌──────────────┐ │ LLM │ └──────┬───────┘ │ ┌─────────┴─────────┐ ▼ ▼ RAG Fine-Tuned Knowledge Model │ │ └─────────┬─────────┘ ▼ Final Answer │ ▼ EVALUATION │ ┌─────────┼─────────┐ ▼ ▼ ▼ Correct? Relevant? Grounded? Don't learn these technologies as isolated topics. Build progressively. Start with an LLM application → add your own knowledge with RAG → adapt the model with fine-tuning → finally build an evaluation layer to measure whether your system actually improved. That progression takes you from "I know how to call an LLM API" to "I can design, improve and evaluate production-style LLM systems."