Building a Multi-Agent AI Pipeline That Ships: LangGraph, RAG, and Evals That Matter

I spent 18 days building an AI product that converts research papers into audience-tailored PowerPoint presentations. Not a toy — a real deployed thing at doc2slides on Railway that anyone can use.

The interesting parts weren't the "make it work" moments. They were the tradeoffs I had to make honestly, and the times I resisted the temptation to add a "clever" fix that would have made things worse.

This post is about those decisions.

Doc2Slides takes a PDF and produces a .pptx

file tailored to four audiences:

The magic is that the same paper produces radically different output based on the audience. A compiler theory paper for a kid becomes "compilers are like magic helpers." The same paper for an executive becomes "advancing compiler technology with formal frameworks."

Code: github.com/manasviboineypally/doc2slides

I built this as a multi-agent pipeline instead of one giant LLM prompt. Here's the flow:

PDF Upload
    ↓
Parser        → extracts sections + metadata
    ↓
Summarizer    → RAG-based section summarization
    ↓
Planner       → designs slide structure for audience
    ↓
Writer        → generates audience-adaptive slide content
    ↓
Builder       → produces editable .pptx file

Each agent is an independent node in a LangGraph state machine. They share a TypedDict

state and read/write specific fields.

Here's what the graph definition actually looks like:

from langgraph.graph import StateGraph, END
from app.agents.state import AgentState
from app.agents.parser import parser_agent
from app.agents.summarizer import summarizer_agent
from app.agents.planner import planner_agent
from app.agents.writer import writer_agent
from app.agents.builder import builder_agent

def build_pipeline():
    graph = StateGraph(AgentState)

    graph.add_node("parser", parser_agent)
    graph.add_node("summarizer", summarizer_agent)
    graph.add_node("planner", planner_agent)
    graph.add_node("writer", writer_agent)
    graph.add_node("builder", builder_agent)

    graph.set_entry_point("parser")
    graph.add_edge("parser", "summarizer")
    graph.add_edge("summarizer", "planner")
    graph.add_edge("planner", "writer")
    graph.add_edge("writer", "builder")
    graph.add_edge("builder", END)

    return graph.compile()

Why LangGraph over a sequential chain? Adding a new agent is a 2-line change to the graph. In a sequential chain, adding a new step often means refactoring the previous ones. State-based multi-agent design scales better.

I built an evaluation harness because I wanted to measure quality, not just claim it. Three eval types:

The parser evals scored 100% (34/34 checks). The summarizer evals averaged 4.4/5.

But the RAG top-1 precision came in at 42%. Only 3 of 7 queries returned the correct section as the top result.

My first instinct: hide the number. Report top-3 (57%) instead.

What I did instead: publish both numbers and explain why.

Looking at the failures revealed a real limitation of RAG:

Methodology

3.6 Stopping Criteria

(a subsection of methodology)Genetic algorithms are discussed in 6 subsections (3.1 through 3.6). Vector search returns the highest-scoring chunk, not the highest-scoring section. For queries about broad topics, subsections often outrank the parent section because they mention the specific term more densely.

This is a known problem in RAG. Solutions include:

None of these are fixed today. But I know exactly what's broken and why — which is more useful than pretending it works.

Lesson: deterministic metrics beat vibes. Vibes let you convince yourself the AI is smart. Metrics tell you where it's dumb.

Users can request any number of slides between 3 and 50. When the paper's actual content density doesn't match the requested slide count, the LLM either pads shallow sections or compresses dense ones. This creates mild redundancy at high slide counts.

The obvious fix: allocate slides based on section word count. Long section = more slides. Short section = fewer slides.

I almost built this. Then I realized: word count is not content density.

Consider:

Word count would systematically reward verbose sections and penalize concise ones. That's not a fix — it's a bug with math.

What I did instead: documented the tradeoff and shipped without the heuristic. From the project's testing_notes.md

:

Rejected quick fix:using section word count as a proxy for content density. Word count is not density — a short section may contain multiple distinct ideas while a long section may ramble around one.

Proper solution deferred:content-aware slide allocation with LLM judgment, verified by an evaluation harness that measures output quality against ground truth. Requires infrastructure work not appropriate for the initial version.

Lesson: the right answer to "should I add this heuristic?" is often "no." Heuristics feel like progress. Sometimes they're anti-progress dressed up as pragmatism.

I built with local SQLite during development but deployed to Railway with PostgreSQL. The migration was one line:

DATABASE_URL = os.getenv("DATABASE_URL")
engine = create_engine(DATABASE_URL, echo=False)

For local dev, .env

has:

DATABASE_URL=sqlite:///./doc2slides.db

For Railway, the environment variable is:

DATABASE_URL=postgresql+psycopg2://postgres:xxx@host:5432/railway

Nothing else changes. SQLAlchemy models are backend-agnostic.

This is boring engineering. But boring engineering is what lets you sleep at night. When someone asks "how do you handle database migrations?" the answer isn't a clever hack — it's "environment-driven configuration and a repository pattern."

The frontend is worth calling out. I used no framework — just HTML, CSS, and vanilla JavaScript in ~500 lines. Zero build step. Anyone can clone the repo, open the file, and understand it in 5 minutes.

For an MVP, that's a feature, not a limitation.

Skipped:

Why: MVP. Every feature has a cost. Shipping the core value (PDF → audience-tailored slides) matters more than shipping every possible feature.

For a portfolio project, "I could have added X but chose not to for these reasons" is a stronger answer than "I added X poorly."

1. Build evals before optimizing. I built the pipeline first, then evals. If I had built evals first, I would have known earlier that my RAG had issues. Now I have to make eval-driven improvements Week 3.

2. Resist heuristics. Every time I thought "this is a quick fix," it was actually a technical debt I was about to bake in. Word count as density. Silent AI slide count overrides. Boolean status flags instead of proper enums.

3. Deploy early. I deployed on Day 16 of 18. I should have deployed on Day 8. Deployment reveals real bugs — environment variable typos, missing dependencies, hardcoded localhost URLs. The sooner you find them, the cheaper they are.

4. Document tradeoffs, not features. Anyone can read code to know what it does. Almost no one leaves notes on why a design choice was made. My testing_notes.md

file is where most of the actual engineering thinking lives.

The project is live, but not "done." Future work:

If you want to try Doc2Slides yourself:

Upload any PDF, pick your audience, get back a deck. Same paper, radically different output depending on who you say you're presenting to.