In this article, you will learn how LangChain, LlamaIndex, and raw API calls each solve a different layer of the LLM application stack, and how to choose among them based on what your project actually requires.
Topics we will cover include:
- What each option is designed to do, stated plainly without marketing spin.
- How the three approaches compare on performance, token overhead, debugging clarity, and code volume.
- A practical decision framework for picking the right level of abstraction before you build — and before that choice becomes expensive to undo.
Let’s not waste any more time.
Introduction #
You have a working prompt. The model is giving good answers. Then the next requirement lands. Maybe it is memory; the model needs to remember what was said three messages ago. Maybe it is retrieval — the model needs to answer questions about documents it was not trained on. Maybe it is tool use; the model needs to check a database, run a calculation, or call an external API before it can respond. Suddenly, a single client.chat.completions.create()
call is not enough, and you are standing at the first real architectural decision in your LLM project.
Three paths exist from that moment: reach for LangChain, reach for LlamaIndex, or build a thin layer on top of the raw SDK yourself. Getting this choice wrong does not break the prototype. It breaks the production system six months later, when you are debugging stack traces 40 frames deep, paying 2.7x what you should be on token costs, or spending a sprint migrating away from breaking API changes.
LLM API spend doubled from $3.5 billion to $8.4 billion between late 2024 and mid-2025. These are real production budgets. The framework layer — the code that sits between your application and the model — directly determines how much of that spend is doing useful work versus paying for abstraction you did not need.
This article gives you an honest comparison: what each option actually is, where it genuinely wins, where it costs you, and a decision framework you can use tomorrow.
The Landscape in Plain English #
Before comparing trade-offs, it helps to understand what each option actually is — not what its marketing says, but what problem it was built to solve.
LangChain started in October 2022 as a general-purpose framework for chaining LLM operations together. Its core idea was that building real applications required composing multiple steps — prompt templates, model calls, output parsers, memory, tools — and there should be a standard way to do that. It has grown into the largest LLM framework by adoption: 119K GitHub stars, 500+ integrations, and a sprawling ecosystem. The LangChain team now builds LangGraph, a separate package for stateful, graph-based agent workflows, as the recommended way to build production agents within the ecosystem.LlamaIndex(launched as GPT Index in November 2022) was built to solve a different problem: getting LLMs to reason over your own data. Its design is organized around data ingestion, chunking, embedding, indexing, and retrieval. Where LangChain is about orchestrating what happens between steps, LlamaIndex is about making the retrieval step itself as accurate and efficient as possible. It sits at 44K GitHub stars with 300+ data connectors through LlamaHub, covering sources like Notion, Google Drive, Slack, PDFs, and databases.Raw API calls means using the OpenAI Python SDK, the Anthropic SDK, or any model provider’s client directly — no orchestration layer, no abstractions beyond what the provider ships. You write the prompt, call the model, and handle the response yourself. This is not the primitive fallback it is sometimes presented as; it is the approach production teams are increasingly migrating back to for workloads where the framework’s complexity stopped paying for itself.
The critical thing to understand before reading any comparison is that these three options are not competing on the same dimension. LangChain is an orchestration toolkit. LlamaIndex is a retrieval toolkit. Raw API calls are a stance on how much abstraction you need. Many production systems use two of them together. The question is always: given what I am actually building, which layer of abstraction earns its cost?
LangChain: The Orchestration Layer #
LangChain’s strength is assembling complexity. If your application involves multiple steps, multiple tools, conditional routing, memory across turns, or agents that reason before acting, LangChain provides the building blocks for all of it, with connectors to 500+ services and a community large enough that someone has already solved most of the edge cases you will encounter.
LangGraph, built by the same team and stable at v1.0 since October 2025, is where the serious agent work lives now. It models agent workflows as directed graphs, where nodes are Python functions, edges are state transitions, and a central typed state object flows through the entire execution. It has built-in persistence via checkpointers to SQLite, PostgreSQL, or Redis, which means agents can mid-workflow, persist their state, and resume hours later. That is genuinely hard to build yourself and is one of LangChain’s clearest justifications in a production context.
The honest trade-offs are worth naming directly. LangChain adds ~10ms framework overhead per step, and LangGraph adds ~14ms. For most human-facing applications that make LLM calls taking 1–3 seconds each, this is irrelevant. For high-throughput pipelines processing thousands of requests per minute, it compounds. Stack traces from LangChain production errors routinely span 15 to 40 frames of internal framework code; finding the actual source of a bug is slower than in a system you wrote yourself. And for simple use cases, one documented comparison found LangChain incurring 2.7x higher costs than a native implementation for a basic RAG pipeline — the abstraction overhead consumed tokens that did not need to be consumed.
LangChain v1.0 (October 2025) committed to API stability after a turbulent v0.1 through v0.3 period that forced multiple breaking migrations. That history is worth knowing. For new projects, the stability concern is largely resolved. For teams running v0.x code in production, the migration cost to v1.0 is real.
Here is a working LangChain LCEL chain — the modern way to compose LangChain operations.
Prerequisites:
pip install langchain langchain-openai python-dotenv
1
pip install langchain langchain-openai python-dotenv
How to run: Save as langchain_chain.py, add OPENAI_API_KEY to your .env, run python langchain_chain.py
import os
from dotenv import load_dotenv
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser
from langchain_openai import ChatOpenAI
load_dotenv()
llm = ChatOpenAI(
model="gpt-4o",
temperature=0.2,
api_key=os.getenv("OPENAI_API_KEY")
)
prompt = ChatPromptTemplate.from_messages([
("system", "You are a concise technical explainer. Keep answers under 100 words."),
("human", "Explain {topic} in simple terms.")
])
parser = StrOutputParser()
chain = prompt | llm | parser
if __name__ == "__main__":
result = chain.invoke({"topic": "vector embeddings"})
print(result)
print("\n--- Streaming response ---")
for chunk in chain.stream({"topic": "RAG pipelines"}):
print(chunk, end="", flush=True)
print()
1234567891011121314151617181920212223242526272829303132333435363738394041424344454647484950515253
What this does: Three objects — prompt
, llm
, parser
— are connected with the |
operator. LangChain’s LCEL executes them in order: the template fills in {topic}
, passes a formatted message to the model, and the parser extracts a plain string from the response. The same chain supports .invoke()
, .stream()
, .batch()
, and .ainvoke()
without any changes to the chain definition itself. That interface consistency is the clearest argument for LangChain on projects that need multiple execution patterns.
Here is the same foundation extended to a tool-using agent with LangGraph.
Prerequisites:
pip install langchain langchain-openai langgraph langchain-community python-dotenv
1
pip install langchain langchain-openai langgraph langchain-community python-dotenv
How to run: Save as langchain_agent.py and run python langchain_agent.py
import os
from dotenv import load_dotenv
from langchain_openai import ChatOpenAI
from langchain.tools import tool
from langchain_community.tools import DuckDuckGoSearchRun
from langchain_core.messages import HumanMessage
from langgraph.prebuilt import create_react_agent
load_dotenv()
llm = ChatOpenAI(model="gpt-4o", temperature=0, api_key=os.getenv("OPENAI_API_KEY"))
search = DuckDuckGoSearchRun()
@tool
def calculate(expression: str) -> str:
"""
Evaluate a safe mathematical expression. Use for arithmetic or percentage calculations.
Input: a Python math expression string (e.g., '1500 * 0.08').
"""
try:
result = eval(expression, {"__builtins__": {}}, {})
return f"Result: {result}"
except Exception as e:
return f"Error: {str(e)}"
tools = [search, calculate]
agent = create_react_agent(llm, tools)
if __name__ == "__main__":
result = agent.invoke({
"messages": [HumanMessage(content="What is 15% of 2400?")]
})
print(result["messages"][-1].content)
12345678910111213141516171819202122232425262728293031323334353637383940414243
What this does: create_react_agent abstracts the full reasoning loop. The model decides whether to use a tool, LangGraph executes the selected tool, feeds the result back into the message history, and repeats until the model has a final answer. What would take 50+ lines in a raw implementation is four lines here. That abstraction is appropriate when you need it. The question the next section addresses is: when do you not?
LlamaIndex: The Retrieval Layer #
LlamaIndex was designed from the ground up for one job: helping LLMs reason over external data. That focus is both its biggest strength and the clearest signal for when to use it. If your application’s central challenge is “how do I get the model to answer accurately from my documents,” LlamaIndex is the right starting point.
The performance numbers reflect that specialization. LlamaIndex indexes documents 2.5x faster than LangChain and hits sub-200ms query latency for 10,000 documents. Its framework overhead of ~6ms compares favorably to LangChain’s ~10ms and LangGraph’s ~14ms. At the token level, LlamaIndex uses ~1.6K tokens per query versus LangChain’s ~2.4K — a 33% difference that adds up quickly at scale.
The architectural reason for those differences is that LlamaIndex treats retrieval as a first-class primitive, not a composable component. Its five core abstractions — data connectors, node parsers, indices, query engines, and workflows — are designed to work together out of the box. Hierarchical chunking preserves parent-child relationships between document sections. Auto-merging retrieval recombines related chunks at query time. Sub-question decomposition breaks complex queries into simpler ones and merges the results. You get all of this with less code: LangChain requires 30–40% more code than LlamaIndex for equivalent RAG pipelines.
Where LlamaIndex is weaker is on the agent side. Its Workflows system handles async, event-driven pipelines well, but stateful multi-turn agents with built-in persistence require more manual implementation than LangGraph. LangGraph’s checkpointing — where an agent s, persists its full state, and resumes later — is something LlamaIndex Workflows can achieve but does not provide out of the box. For document Q&A and knowledge retrieval, this rarely matters. For long-running agentic workflows with human-in-the-loop requirements, it matters a great deal.
Here is a complete LlamaIndex RAG pipeline, from document ingestion to query.
Prerequisites:
pip install llama-index llama-index-llms-openai llama-index-embeddings-openai python-dotenv
1
pip install llama-index llama-index-llms-openai llama-index-embeddings-openai python-dotenv
How to run: Save as llamaindex_rag.py and run python llamaindex_rag.py
import os
from dotenv import load_dotenv
from llama_index.core import VectorStoreIndex, Document, Settings
from llama_index.llms.openai import OpenAI as LlamaOpenAI
from llama_index.embeddings.openai import OpenAIEmbedding
load_dotenv()
Settings.llm = LlamaOpenAI(
model="gpt-4o",
temperature=0,
api_key=os.getenv("OPENAI_API_KEY")
)
Settings.embed_model = OpenAIEmbedding(
model="text-embedding-3-small", # Fast and cost-effective for most RAG tasks
api_key=os.getenv("OPENAI_API_KEY")
)
documents = [
Document(
text=(
"LlamaIndex is a data framework for LLM applications. "
"It specializes in document ingestion, chunking, embedding, and retrieval. "
"Core abstractions: data connectors, node parsers, indices, query engines, "
"and workflows. LlamaHub provides 300+ pre-built data connectors."
),
metadata={"source": "llamaindex_overview"}
),
Document(
text=(
"LangChain is a general-purpose LLM orchestration framework. "
"It excels at chaining operations, multi-step agents, tool use, and memory. "
"LangGraph -- the recommended way to build stateful agents in the LangChain "
"ecosystem -- stabilized at v1.0 in October 2025."
),
metadata={"source": "langchain_overview"}
),
Document(
text=(
"Raw API calls use the OpenAI or Anthropic SDK directly with no framework. "
"This approach has the lowest latency and highest transparency. "
"Best for simple, one-off tasks where framework abstraction adds no value. "
"As complexity grows, a thin internal wrapper is usually preferable to "
"adopting a full orchestration framework."
),
metadata={"source": "raw_api_overview"}
),
]
index = VectorStoreIndex.from_documents(documents)
query_engine = index.as_query_engine(
similarity_top_k=2,
response_mode="compact"
)
if __name__ == "__main__":
questions = [
"What is LlamaIndex best suited for?",
"How does LangChain differ from LlamaIndex?",
"When should I use raw API calls instead of a framework?",
]
for q in questions:
print(f"Q: {q}")
response = query_engine.query(q)
print(f"A: {response}\n")
12345678910111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455565758596061626364656667686970717273747576777879808182838485868788899091
What this does: Settings.llm
and Settings.embed_model
configure the entire pipeline once. VectorStoreIndex.from_documents()
handles chunking, embedding, and indexing in a single call — a process that takes 30–40% more code in LangChain. as_query_engine()
then creates a retrieval + generation pipeline with two lines. The similarity_top_k
and response_mode
parameters give you control over the retrieval behavior without requiring you to assemble the retrieval components yourself. That is the LlamaIndex value proposition in concrete form: less assembly, more retrieval quality.
Raw API Calls: The Minimal Path #
The default assumption in most LLM developer communities is that you start with raw API calls and graduate to a framework as your project grows. The pattern worth examining in 2026 is the reverse: teams that started with LangChain and are quietly rewriting to raw SDKs.
The OpenAI Agents SDK, released in March 2025 with 26,900 GitHub stars and 10.3 million monthly downloads, provides tool use, multi-agent handoffs, built-in tracing, and guardrails in a minimal package. Its overhead per tool call is 2–5ms versus LangChain’s 10–30ms. Teams migrating from LangChain to raw SDKs typically see a 40–60% reduction in code volume and a 70–90% reduction in monthly framework maintenance burden.
The argument for the raw path is not that frameworks are bad. It is that the value of an abstraction layer depends entirely on whether it is hiding complexity you actually face. In 2022, building prompt chains and handling tool calls reliably required framework support because vendor APIs were inconsistent. By 2026, OpenAI and Anthropic have absorbed tool calling, streaming, function schemas, and multi-turn memory into their native SDKs. The framework’s abstractions no longer hide meaningful differences. They hide clarity.
Raw API is consistently the fastest option, with no framework overhead and no extra LLM calls for orchestration. Frameworks add 100–500ms of Python overhead per agent step. For latency-sensitive workloads — real-time customer support, voice agents, and high-throughput pipelines — that overhead is real and worth avoiding.
Here is a complete tool-using agent built on the raw OpenAI SDK in under 80 lines.
Prerequisites:
pip install openai python-dotenv
1
pip install openai python-dotenv
How to run: Save as raw_api_agent.py and run python raw_api_agent.py
import os
import json
from dotenv import load_dotenv
from openai import OpenAI
load_dotenv()
client = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))
TOOLS = [
{
"type": "function",
"function": {
"name": "calculate",
"description": (
"Evaluate a mathematical expression. Use for arithmetic, "
"percentages, or numerical computation. "
"Input: a Python math expression as a string."
),
"parameters": {
"type": "object",
"properties": {
"expression": {
"type": "string",
"description": "A Python math expression, e.g. '1500 * 0.08'"
}
},
"required": ["expression"]
}
}
},
{
"type": "function",
"function": {
"name": "get_word_count",
"description": "Count the number of words in a given string of text.",
"parameters": {
"type": "object",
"properties": {
"text": {"type": "string", "description": "The text to count."}
},
"required": ["text"]
}
}
}
]
def calculate(expression: str) -> str:
try:
result = eval(expression, {"__builtins__": {}}, {})
return str(result)
except Exception as e:
return f"Error: {e}"
def get_word_count(text: str) -> str:
return str(len(text.split()))
TOOL_DISPATCH = {"calculate": calculate, "get_word_count": get_word_count}
def run_agent(user_message: str) -> str:
"""
A complete ReAct-style agent loop using raw OpenAI tool calls.
The model decides whether to call a tool or return a final answer.
The loop continues until the model stops requesting tool calls.
Every step is visible -- no framework wrapping, no hidden logic.
"""
messages = [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": user_message},
]
while True:
response = client.chat.completions.create(
model="gpt-4o",
messages=messages,
tools=TOOLS,
tool_choice="auto", # Model decides: call a tool or respond directly
temperature=0,
)
message = response.choices[0].message
messages.append(message) # Always add the assistant message to history
if not message.tool_calls:
return message.content
for tool_call in message.tool_calls:
name = tool_call.function.name
args = json.loads(tool_call.function.arguments)
fn = TOOL_DISPATCH.get(name)
result = fn(**args) if fn else f"Unknown tool: {name}"
messages.append({
"role": "tool",
"tool_call_id": tool_call.id,
"content": result,
})
if __name__ == "__main__":
queries = [
"What is 18% of 3500?",
"How many words are in: The quick brown fox jumps over the lazy dog?",
"Split 240 items into groups of 16. How many groups?",
]
for q in queries:
print(f"Q: {q}\nA: {run_agent(q)}\n")
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123
What this does: The agent loop is fully transparent. There is no framework between you and the model’s response. The while True loop runs until message.tool_calls
is empty, which happens when the model decides it has enough information to answer directly. Every message — system, user, assistant, and tool result — is in a plain Python list you can inspect, log, or modify at any point. That transparency is the raw path’s core advantage: when something breaks, you know exactly where to look.
Head-to-Head Comparison #
The same task was evaluated across three dimensions. All measurements reflect current benchmarks from independent analysis cited throughout this article.
Framework Overhead and Performance
Metric | Raw API | LlamaIndex | LangChain (LCEL) | LangGraph | |---|---|---|---|---| | Framework overhead | ~0ms | ~6ms | ~10ms | ~14ms | | Token overhead (per query) | 0 | ~1.6K | ~2.4K | ~2.0K | | Per tool call latency | 2–5ms | N/A | 10–30ms | 10–30ms | | Stack trace depth on error | 2–5 frames | 5–10 frames | 15–40 frames | 15–40 frames | | Debug transparency | High | Medium | Low | Low |
Code Volume: Same RAG Task, Three Ways
This is the most concrete way to feel the trade-off. All three implementations below answer the same question from the same context document:
Implementation | Lines of code | Framework install size | Debugging clarity | |---|---|---|---| | Raw OpenAI SDK | ~20 lines | openai only | Full visibility | | LlamaIndex | ~15 lines | llama-index + plugins | Medium | | LangChain LCEL | ~18 lines | langchain + langchain-openai | Low–medium |
For a basic one-document Q&A, the difference is marginal. Where LlamaIndex’s code advantage compounds is when you add chunking strategies, multiple documents, re-ranking, metadata filtering, and hybrid search — each of which requires more assembly in LangChain than in LlamaIndex.
When Each One Breaks
Knowing when each approach fails is as useful as knowing when it succeeds.
Failure mode | Raw API | LlamaIndex | LangChain | |---|---|---|---| | Retrieval accuracy degrades | You built it, you fix it | Tune chunking/index strategy | Tune each pipeline component separately | | Agent loops indefinitely | Add max_iterations manually | Workflow timeout | max_iterations parameter | | Prompt changes break output | Immediate, obvious | Immediate, obvious | May propagate through chain silently | | Model API changes | Update SDK | Update llama-index package | Update langchain-openai + retest | | Debugging a production error | Direct, small stack | Moderate | Deep stack traces, hard to isolate | | Scaling to high throughput | Optimal | Good | Framework overhead compounds |
Full Working Example #
The same document Q&A task implemented three ways. Same input document, same question, different path through the stack. Read these side by side and the trade-offs become concrete.
Prerequisites:
pip install openai langchain langchain-openai llama-index \
llama-index-llms-openai llama-index-embeddings-openai python-dotenv
12
pip install openai langchain langchain-openai llama-index \ llama-index-llms-openai llama-index-embeddings-openai python-dotenv
How to run: Save as three_ways.py and run python three_ways.py
import os
import time
from dotenv import load_dotenv
load_dotenv()
QUESTION = "What is retrieval-augmented generation and why does it matter?"
CONTEXT_DOC = """
Retrieval-Augmented Generation (RAG) is a technique that improves LLM responses
by fetching relevant context from an external knowledge base before generating
an answer. Instead of relying solely on training data, RAG retrieves the most
relevant document chunks and includes them in the prompt. This reduces
hallucinations, keeps answers grounded in your actual data, and allows the model
to answer questions about information it was never trained on.
"""
def raw_api_answer(question: str, context: str) -> str:
"""Answer a question using context, via raw OpenAI SDK -- no framework."""
from openai import OpenAI
client = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))
response = client.chat.completions.create(
model="gpt-4o",
temperature=0,
messages=[
{
"role": "system",
"content": (
"Answer questions using only the provided context. "
"If the answer is not in the context, say so clearly."
)
},
{
"role": "user",
"content": f"Context:\n{context}\n\nQuestion: {question}"
}
]
)
return response.choices[0].message.content
def llamaindex_answer(question: str, context: str) -> str:
"""Answer a question using LlamaIndex -- purpose-built retrieval pipeline."""
from llama_index.core import VectorStoreIndex, Document, Settings
from llama_index.llms.openai import OpenAI as LlamaOpenAI
from llama_index.embeddings.openai import OpenAIEmbedding
Settings.llm = LlamaOpenAI(
model="gpt-4o", temperature=0,
api_key=os.getenv("OPENAI_API_KEY")
)
Settings.embed_model = OpenAIEmbedding(
model="text-embedding-3-small",
api_key=os.getenv("OPENAI_API_KEY")
)
index = VectorStoreIndex.from_documents([Document(text=context)])
query_engine = index.as_query_engine(similarity_top_k=1)
return str(query_engine.query(question))
def langchain_answer(question: str, context: str) -> str:
"""Answer a question using a LangChain LCEL chain."""
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser
from langchain_openai import ChatOpenAI
llm = ChatOpenAI(
model="gpt-4o", temperature=0,
api_key=os.getenv("OPENAI_API_KEY")
)
prompt = ChatPromptTemplate.from_messages([
("system",
"Answer using only the provided context. "
"If the answer is not in the context, say so.\n\nContext:\n{context}"),
("human", "{question}")
])
chain = prompt | llm | StrOutputParser()
return chain.invoke({"context": context, "question": question})
if __name__ == "__main__":
approaches = [
("Raw OpenAI SDK", raw_api_answer),
("LlamaIndex", llamaindex_answer),
("LangChain LCEL", langchain_answer),
]
for name, fn in approaches:
print(f"\n{'='*60}")
print(f"Approach: {name}")
print(f"{'='*60}")
start = time.perf_counter()
answer = fn(QUESTION, CONTEXT_DOC)
elapsed = time.perf_counter() - start
print(f"Answer: {answer}")
print(f"Time (excluding LLM): visible in wall clock")
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132
What this does: All three functions receive the same QUESTION
and CONTEXT_DOC
and return a string answer. The raw API version manually constructs the message list and extracts the response. The LlamaIndex version uses from_documents()
and as_query_engine()
to handle the pipeline. The LangChain version assembles a prompt, model, and parser with the |
operator. At this scale — one document, one question — the differences are minimal. Feed this function 500 documents and a complex query, and the gap between LlamaIndex’s purpose-built retrieval and the other two approaches opens up significantly.
Wrapping Up #
The framework decision is not about which option has the most GitHub stars or the most features. It is about matching the abstraction level of your tool to the actual complexity of your problem.
For simple, one-shot tasks, raw API calls are faster to write, faster to run, and easier to debug than any framework. For document retrieval at any meaningful scale, LlamaIndex earns its dependency through better chunking, faster indexing, and less code. For stateful agents with memory, tools, and multi-step reasoning, LangGraph’s persistence and graph-based control flow are genuinely hard to replicate cleanly with a hand-rolled loop.
The pattern that most production teams converge on by mid-2026 is not a single framework but a layered stack: raw SDK for the simple calls, LlamaIndex for the retrieval layer, LangGraph for the agent loop, and LangSmith for tracing across everything. None of those choices locks you out of the others. They compose.
The practical rule is this: start with the minimal option that handles your current requirements, and add a framework when you hit a problem the framework was built to solve — not before. A retrieval problem you encounter is a reason to add LlamaIndex. A state management problem you encounter is a reason to add LangGraph. Adding either before you feel the pain they address means adding maintenance overhead for a future problem that may not arrive in the shape you expected.