LLM Orchestration Frameworks Compared: LangChain vs. LlamaIndex vs. Raw API Calls

LangChain, LlamaIndex, and raw API calls each address different layers of the LLM application stack, with LangChain focusing on general orchestration, LlamaIndex on data retrieval, and raw calls offering minimal abstraction. The choice between them directly impacts production costs and debugging complexity, as LLM API spend doubled from $3.5 billion to $8.4 billion between late 2024 and mid-2025.

In this article, you will learn how LangChain, LlamaIndex, and raw API calls each solve a different layer of the LLM application stack, and how to choose among them based on what your project actually requires. Topics we will cover include: - What each option is designed to do, stated plainly without marketing spin. - How the three approaches compare on performance, token overhead, debugging clarity, and code volume. - A practical decision framework for picking the right level of abstraction before you build — and before that choice becomes expensive to undo. Let’s not waste any more time. Introduction You have a working prompt. The model is giving good answers. Then the next requirement lands. Maybe it is memory; the model needs to remember what was said three messages ago. Maybe it is retrieval — the model needs to answer questions about documents it was not trained on. Maybe it is tool use; the model needs to check a database, run a calculation, or call an external API before it can respond. Suddenly, a single client.chat.completions.create call is not enough, and you are standing at the first real architectural decision in your LLM project. Three paths exist from that moment: reach for LangChain, reach for LlamaIndex, or build a thin layer on top of the raw SDK yourself. Getting this choice wrong does not break the prototype. It breaks the production system six months later, when you are debugging stack traces 40 frames deep, paying 2.7x what you should be on token costs, or spending a sprint migrating away from breaking API changes. LLM API spend doubled from \$3.5 billion to \$8.4 billion between late 2024 and mid-2025 https://www.getmaxim.ai/articles/best-llm-cost-tracking-tools-in-2026/ . These are real production budgets. The framework layer — the code that sits between your application and the model — directly determines how much of that spend is doing useful work versus paying for abstraction you did not need. This article gives you an honest comparison: what each option actually is, where it genuinely wins, where it costs you, and a decision framework you can use tomorrow. The Landscape in Plain English Before comparing trade-offs, it helps to understand what each option actually is — not what its marketing says, but what problem it was built to solve. LangChain started in October 2022 as a general-purpose framework for chaining LLM operations together. Its core idea was that building real applications required composing multiple steps — prompt templates, model calls, output parsers, memory, tools — and there should be a standard way to do that. It has grown into the largest LLM framework by adoption: 119K GitHub stars, 500+ integrations, and a sprawling ecosystem. The LangChain team now builds LangGraph, a separate package for stateful, graph-based agent workflows, as the recommended way to build production agents within the ecosystem. LlamaIndex launched as GPT Index in November 2022 was built to solve a different problem: getting LLMs to reason over your own data. Its design is organized around data ingestion, chunking, embedding, indexing, and retrieval. Where LangChain is about orchestrating what happens between steps, LlamaIndex is about making the retrieval step itself as accurate and efficient as possible. It sits at 44K GitHub stars with 300+ data connectors through LlamaHub, covering sources like Notion, Google Drive, Slack, PDFs, and databases. Raw API calls means using the OpenAI Python SDK, the Anthropic SDK, or any model provider’s client directly — no orchestration layer, no abstractions beyond what the provider ships. You write the prompt, call the model, and handle the response yourself. This is not the primitive fallback it is sometimes presented as; it is the approach production teams are increasingly migrating back to for workloads where the framework’s complexity stopped paying for itself. The critical thing to understand before reading any comparison is that these three options are not competing on the same dimension. LangChain is an orchestration toolkit. LlamaIndex is a retrieval toolkit. Raw API calls are a stance on how much abstraction you need. Many production systems use two of them together. The question is always: given what I am actually building, which layer of abstraction earns its cost? LangChain: The Orchestration Layer LangChain’s strength is assembling complexity. If your application involves multiple steps, multiple tools, conditional routing, memory across turns, or agents that reason before acting, LangChain provides the building blocks for all of it, with connectors to 500+ services and a community large enough that someone has already solved most of the edge cases you will encounter. LangGraph, built by the same team and stable at v1.0 since October 2025, is where the serious agent work lives now. It models agent workflows as directed graphs, where nodes are Python functions, edges are state transitions, and a central typed state object flows through the entire execution. It has built-in persistence via checkpointers to SQLite, PostgreSQL, or Redis https://www.morphllm.com/comparisons/langchain-vs-llamaindex , which means agents can pause mid-workflow, persist their state, and resume hours later. That is genuinely hard to build yourself and is one of LangChain’s clearest justifications in a production context. The honest trade-offs are worth naming directly. LangChain adds ~10ms framework overhead per step, and LangGraph adds ~14ms https://aimultiple.com/rag-frameworks . For most human-facing applications that make LLM calls taking 1–3 seconds each, this is irrelevant. For high-throughput pipelines processing thousands of requests per minute, it compounds. Stack traces from LangChain production errors routinely span 15 to 40 frames of internal framework code https://ravoid.com/blog/langchain-exit-raw-sdk-migration-2026 ; finding the actual source of a bug is slower than in a system you wrote yourself. And for simple use cases, one documented comparison found LangChain incurring 2.7x higher costs than a native implementation for a basic RAG pipeline https://checkthat.ai/brands/langchain/pricing — the abstraction overhead consumed tokens that did not need to be consumed. LangChain v1.0 October 2025 committed to API stability after a turbulent v0.1 through v0.3 period that forced multiple breaking migrations. That history is worth knowing. For new projects, the stability concern is largely resolved. For teams running v0.x code in production, the migration cost to v1.0 is real. Here is a working LangChain LCEL chain — the modern way to compose LangChain operations. Prerequisites: pip install langchain langchain-openai python-dotenv 1 pip install langchain langchain-openai python-dotenv How to run: Save as langchain chain.py , add OPENAI API KEY to your .env , run python langchain chain.py langchain chain.py A LangChain LCEL chain: prompt template → model → output parser Prerequisites: pip install langchain langchain-openai python-dotenv How to run: python langchain chain.py import os from dotenv import load dotenv from langchain core.prompts import ChatPromptTemplate from langchain core.output parsers import StrOutputParser from langchain openai import ChatOpenAI load dotenv ── MODEL ───────────────────────────────────────────────────────────────────── ChatOpenAI wraps OpenAI's chat models. Swap the model string to switch to gpt-4o-mini cheaper or claude-3-5-sonnet via langchain-anthropic -- the chain code below stays identical either way. This model portability is one of LangChain's genuine advantages over raw API calls. llm = ChatOpenAI model="gpt-4o", temperature=0.2, api key=os.getenv "OPENAI API KEY" ── PROMPT TEMPLATE ─────────────────────────────────────────────────────────── ChatPromptTemplate defines the message structure with named variables. {topic} gets filled in at runtime -- templates are reusable and versionable. prompt = ChatPromptTemplate.from messages "system", "You are a concise technical explainer. Keep answers under 100 words." , "human", "Explain {topic} in simple terms." ── OUTPUT PARSER ───────────────────────────────────────────────────────────── StrOutputParser extracts the text content from the model's AIMessage response. Without it you get back an AIMessage object rather than a plain string. parser = StrOutputParser ── CHAIN LCEL ────────────────────────────────────────────────────────────── The pipe operator | builds a sequential chain: prompt → llm → parser. LCEL LangChain Expression Language makes the composition readable and supports streaming, batching, and async execution with the same interface. chain = prompt | llm | parser if name == " main ": invoke runs the full chain synchronously result = chain.invoke {"topic": "vector embeddings"} print result stream yields tokens as they arrive -- no code changes needed for streaming print "\n--- Streaming response ---" for chunk in chain.stream {"topic": "RAG pipelines"} : print chunk, end="", flush=True print 1234567891011121314151617181920212223242526272829303132333435363738394041424344454647484950515253 langchain chain.py A LangChain LCEL chain: prompt template → model → output parser Prerequisites: pip install langchain langchain-openai python-dotenv How to run: python langchain chain.py import osfrom dotenv import load dotenvfrom langchain core.prompts import ChatPromptTemplatefrom langchain core.output parsers import StrOutputParserfrom langchain openai import ChatOpenAI load dotenv ── MODEL ───────────────────────────────────────────────────────────────────── ChatOpenAI wraps OpenAI's chat models. Swap the model string to switch to gpt-4o-mini cheaper or claude-3-5-sonnet via langchain-anthropic -- the chain code below stays identical either way. This model portability is one of LangChain's genuine advantages over raw API calls.llm = ChatOpenAI model="gpt-4o", temperature=0.2, api key=os.getenv "OPENAI API KEY" ── PROMPT TEMPLATE ─────────────────────────────────────────────────────────── ChatPromptTemplate defines the message structure with named variables. {topic} gets filled in at runtime -- templates are reusable and versionable.prompt = ChatPromptTemplate.from messages "system", "You are a concise technical explainer. Keep answers under 100 words." , "human", "Explain {topic} in simple terms." ── OUTPUT PARSER ───────────────────────────────────────────────────────────── StrOutputParser extracts the text content from the model's AIMessage response. Without it you get back an AIMessage object rather than a plain string.parser = StrOutputParser ── CHAIN LCEL ────────────────────────────────────────────────────────────── The pipe operator | builds a sequential chain: prompt → llm → parser. LCEL LangChain Expression Language makes the composition readable and supports streaming, batching, and async execution with the same interface.chain = prompt | llm | parser if name == " main ": invoke runs the full chain synchronously result = chain.invoke {"topic": "vector embeddings"} print result stream yields tokens as they arrive -- no code changes needed for streaming print "\n--- Streaming response ---" for chunk in chain.stream {"topic": "RAG pipelines"} : print chunk, end="", flush=True print What this does: Three objects — prompt , llm , parser — are connected with the | operator. LangChain’s LCEL executes them in order: the template fills in {topic} , passes a formatted message to the model, and the parser extracts a plain string from the response. The same chain supports .invoke , .stream , .batch , and .ainvoke without any changes to the chain definition itself. That interface consistency is the clearest argument for LangChain on projects that need multiple execution patterns. Here is the same foundation extended to a tool-using agent with LangGraph. Prerequisites: pip install langchain langchain-openai langgraph langchain-community python-dotenv 1 pip install langchain langchain-openai langgraph langchain-community python-dotenv How to run: Save as langchain agent.py and run python langchain agent.py langchain agent.py A LangGraph ReAct agent with two tools: web search and a calculator Prerequisites: pip install langchain langchain-openai langgraph langchain-community python-dotenv How to run: python langchain agent.py import os from dotenv import load dotenv from langchain openai import ChatOpenAI from langchain.tools import tool from langchain community.tools import DuckDuckGoSearchRun from langchain core.messages import HumanMessage from langgraph.prebuilt import create react agent load dotenv llm = ChatOpenAI model="gpt-4o", temperature=0, api key=os.getenv "OPENAI API KEY" Web search -- no API key required search = DuckDuckGoSearchRun @tool def calculate expression: str - str: """ Evaluate a safe mathematical expression. Use for arithmetic or percentage calculations. Input: a Python math expression string e.g., '1500 0.08' . """ try: result = eval expression, {" builtins ": {}}, {} return f"Result: {result}" except Exception as e: return f"Error: {str e }" tools = search, calculate create react agent wires together the LLM, tools, and a built-in ReAct loop. The agent thinks, calls a tool, reads the result, and continues until done. agent = create react agent llm, tools if name == " main ": result = agent.invoke { "messages": HumanMessage content="What is 15% of 2400?" } print result "messages" -1 .content 12345678910111213141516171819202122232425262728293031323334353637383940414243 langchain agent.py A LangGraph ReAct agent with two tools: web search and a calculator Prerequisites: pip install langchain langchain-openai langgraph langchain-community python-dotenv How to run: python langchain agent.py import osfrom dotenv import load dotenvfrom langchain openai import ChatOpenAIfrom langchain.tools import toolfrom langchain community.tools import DuckDuckGoSearchRunfrom langchain core.messages import HumanMessagefrom langgraph.prebuilt import create react agent load dotenv llm = ChatOpenAI model="gpt-4o", temperature=0, api key=os.getenv "OPENAI API KEY" Web search -- no API key requiredsearch = DuckDuckGoSearchRun @tooldef calculate expression: str - str: """ Evaluate a safe mathematical expression. Use for arithmetic or percentage calculations. Input: a Python math expression string e.g., '1500 0.08' . """ try: result = eval expression, {" builtins ": {}}, {} return f"Result: {result}" except Exception as e: return f"Error: {str e }" tools = search, calculate create react agent wires together the LLM, tools, and a built-in ReAct loop. The agent thinks, calls a tool, reads the result, and continues until done.agent = create react agent llm, tools if name == " main ": result = agent.invoke { "messages": HumanMessage content="What is 15% of 2400?" } print result "messages" -1 .content What this does: create react agent abstracts the full reasoning loop. The model decides whether to use a tool, LangGraph executes the selected tool, feeds the result back into the message history, and repeats until the model has a final answer. What would take 50+ lines in a raw implementation is four lines here. That abstraction is appropriate when you need it. The question the next section addresses is: when do you not? LlamaIndex: The Retrieval Layer LlamaIndex was designed from the ground up for one job: helping LLMs reason over external data. That focus is both its biggest strength and the clearest signal for when to use it. If your application’s central challenge is “how do I get the model to answer accurately from my documents,” LlamaIndex is the right starting point. The performance numbers reflect that specialization. LlamaIndex indexes documents 2.5x faster than LangChain and hits sub-200ms query latency for 10,000 documents https://wifitalents.com/llamaindex-statistics/ . Its framework overhead of ~6ms compares favorably to LangChain’s ~10ms and LangGraph’s ~14ms https://aimultiple.com/rag-frameworks . At the token level, LlamaIndex uses ~1.6K tokens per query versus LangChain’s ~2.4K https://www.morphllm.com/comparisons/langchain-vs-llamaindex — a 33% difference that adds up quickly at scale. The architectural reason for those differences is that LlamaIndex treats retrieval as a first-class primitive, not a composable component. Its five core abstractions — data connectors, node parsers, indices, query engines, and workflows — are designed to work together out of the box. Hierarchical chunking preserves parent-child relationships between document sections. Auto-merging retrieval recombines related chunks at query time. Sub-question decomposition breaks complex queries into simpler ones and merges the results. You get all of this with less code: LangChain requires 30–40% more code than LlamaIndex for equivalent RAG pipelines https://www.morphllm.com/comparisons/langchain-vs-llamaindex . Where LlamaIndex is weaker is on the agent side. Its Workflows system handles async, event-driven pipelines well, but stateful multi-turn agents with built-in persistence require more manual implementation than LangGraph. LangGraph’s checkpointing — where an agent pauses, persists its full state, and resumes later — is something LlamaIndex Workflows can achieve but does not provide out of the box. For document Q&A and knowledge retrieval, this rarely matters. For long-running agentic workflows with human-in-the-loop requirements, it matters a great deal. Here is a complete LlamaIndex RAG pipeline, from document ingestion to query. Prerequisites: pip install llama-index llama-index-llms-openai llama-index-embeddings-openai python-dotenv 1 pip install llama-index llama-index-llms-openai llama-index-embeddings-openai python-dotenv How to run: Save as llamaindex rag.py and run python llamaindex rag.py llamaindex rag.py Complete LlamaIndex RAG pipeline: ingest documents → index → query Prerequisites: pip install llama-index llama-index-llms-openai llama-index-embeddings-openai python-dotenv How to run: python llamaindex rag.py import os from dotenv import load dotenv from llama index.core import VectorStoreIndex, Document, Settings from llama index.llms.openai import OpenAI as LlamaOpenAI from llama index.embeddings.openai import OpenAIEmbedding load dotenv ── GLOBAL SETTINGS ─────────────────────────────────────────────────────────── LlamaIndex v0.10+ uses a global Settings object instead of ServiceContext. Configure your LLM and embedding model once here -- all pipeline components pick them up automatically. Swap models here to change the whole pipeline. Settings.llm = LlamaOpenAI model="gpt-4o", temperature=0, api key=os.getenv "OPENAI API KEY" Settings.embed model = OpenAIEmbedding model="text-embedding-3-small", Fast and cost-effective for most RAG tasks api key=os.getenv "OPENAI API KEY" ── DOCUMENTS ───────────────────────────────────────────────────────────────── In production, replace with: SimpleDirectoryReader "./docs" .load data LlamaHub provides 300+ connectors for Notion, Google Drive, PDFs, databases. Documents created inline here to keep the example fully self-contained. documents = Document text= "LlamaIndex is a data framework for LLM applications. " "It specializes in document ingestion, chunking, embedding, and retrieval. " "Core abstractions: data connectors, node parsers, indices, query engines, " "and workflows. LlamaHub provides 300+ pre-built data connectors." , metadata={"source": "llamaindex overview"} , Document text= "LangChain is a general-purpose LLM orchestration framework. " "It excels at chaining operations, multi-step agents, tool use, and memory. " "LangGraph -- the recommended way to build stateful agents in the LangChain " "ecosystem -- stabilized at v1.0 in October 2025." , metadata={"source": "langchain overview"} , Document text= "Raw API calls use the OpenAI or Anthropic SDK directly with no framework. " "This approach has the lowest latency and highest transparency. " "Best for simple, one-off tasks where framework abstraction adds no value. " "As complexity grows, a thin internal wrapper is usually preferable to " "adopting a full orchestration framework." , metadata={"source": "raw api overview"} , ── INDEX ───────────────────────────────────────────────────────────────────── from documents handles the full pipeline: chunk → embed → store. By default, vectors are stored in memory. For production, pass a vector store: index = VectorStoreIndex.from documents docs, storage context=storage context where storage context points to Pinecone, Weaviate, Chroma, etc. index = VectorStoreIndex.from documents documents ── QUERY ENGINE ────────────────────────────────────────────────────────────── as query engine creates a retrieval + generation pipeline in one call. similarity top k=2 retrieves the 2 most relevant chunks per query. response mode="compact" merges retrieved chunks before passing to the LLM -- reduces token usage compared to "default" mode, which sends each chunk separately. query engine = index.as query engine similarity top k=2, response mode="compact" if name == " main ": questions = "What is LlamaIndex best suited for?", "How does LangChain differ from LlamaIndex?", "When should I use raw API calls instead of a framework?", for q in questions: print f"Q: {q}" response = query engine.query q print f"A: {response}\n" 12345678910111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455565758596061626364656667686970717273747576777879808182838485868788899091 llamaindex rag.py Complete LlamaIndex RAG pipeline: ingest documents → index → query Prerequisites: pip install llama-index llama-index-llms-openai llama-index-embeddings-openai python-dotenv How to run: python llamaindex rag.py import osfrom dotenv import load dotenvfrom llama index.core import VectorStoreIndex, Document, Settingsfrom llama index.llms.openai import OpenAI as LlamaOpenAIfrom llama index.embeddings.openai import OpenAIEmbedding load dotenv ── GLOBAL SETTINGS ─────────────────────────────────────────────────────────── LlamaIndex v0.10+ uses a global Settings object instead of ServiceContext. Configure your LLM and embedding model once here -- all pipeline components pick them up automatically. Swap models here to change the whole pipeline.Settings.llm = LlamaOpenAI model="gpt-4o", temperature=0, api key=os.getenv "OPENAI API KEY" Settings.embed model = OpenAIEmbedding model="text-embedding-3-small", Fast and cost-effective for most RAG tasks api key=os.getenv "OPENAI API KEY" ── DOCUMENTS ───────────────────────────────────────────────────────────────── In production, replace with: SimpleDirectoryReader "./docs" .load data LlamaHub provides 300+ connectors for Notion, Google Drive, PDFs, databases. Documents created inline here to keep the example fully self-contained.documents = Document text= "LlamaIndex is a data framework for LLM applications. " "It specializes in document ingestion, chunking, embedding, and retrieval. " "Core abstractions: data connectors, node parsers, indices, query engines, " "and workflows. LlamaHub provides 300+ pre-built data connectors." , metadata={"source": "llamaindex overview"} , Document text= "LangChain is a general-purpose LLM orchestration framework. " "It excels at chaining operations, multi-step agents, tool use, and memory. " "LangGraph -- the recommended way to build stateful agents in the LangChain " "ecosystem -- stabilized at v1.0 in October 2025." , metadata={"source": "langchain overview"} , Document text= "Raw API calls use the OpenAI or Anthropic SDK directly with no framework. " "This approach has the lowest latency and highest transparency. " "Best for simple, one-off tasks where framework abstraction adds no value. " "As complexity grows, a thin internal wrapper is usually preferable to " "adopting a full orchestration framework." , metadata={"source": "raw api overview"} , ── INDEX ───────────────────────────────────────────────────────────────────── from documents handles the full pipeline: chunk → embed → store. By default, vectors are stored in memory. For production, pass a vector store: index = VectorStoreIndex.from documents docs, storage context=storage context where storage context points to Pinecone, Weaviate, Chroma, etc.index = VectorStoreIndex.from documents documents ── QUERY ENGINE ────────────────────────────────────────────────────────────── as query engine creates a retrieval + generation pipeline in one call. similarity top k=2 retrieves the 2 most relevant chunks per query. response mode="compact" merges retrieved chunks before passing to the LLM -- reduces token usage compared to "default" mode, which sends each chunk separately.query engine = index.as query engine similarity top k=2, response mode="compact" if name == " main ": questions = "What is LlamaIndex best suited for?", "How does LangChain differ from LlamaIndex?", "When should I use raw API calls instead of a framework?", for q in questions: print f"Q: {q}" response = query engine.query q print f"A: {response}\n" What this does: Settings.llm and Settings.embed model configure the entire pipeline once. VectorStoreIndex.from documents handles chunking, embedding, and indexing in a single call — a process that takes 30–40% more code in LangChain. as query engine then creates a retrieval + generation pipeline with two lines. The similarity top k and response mode parameters give you control over the retrieval behavior without requiring you to assemble the retrieval components yourself. That is the LlamaIndex value proposition in concrete form: less assembly, more retrieval quality. Raw API Calls: The Minimal Path The default assumption in most LLM developer communities is that you start with raw API calls and graduate to a framework as your project grows. The pattern worth examining in 2026 is the reverse: teams that started with LangChain and are quietly rewriting to raw SDKs. The OpenAI Agents SDK, released in March 2025 with 26,900 GitHub stars and 10.3 million monthly downloads, provides tool use, multi-agent handoffs, built-in tracing, and guardrails in a minimal package. Its overhead per tool call is 2–5ms versus LangChain’s 10–30ms. Teams migrating from LangChain to raw SDKs typically see a 40–60% reduction in code volume and a 70–90% reduction in monthly framework maintenance burden. The argument for the raw path is not that frameworks are bad. It is that the value of an abstraction layer depends entirely on whether it is hiding complexity you actually face. In 2022, building prompt chains and handling tool calls reliably required framework support because vendor APIs were inconsistent. By 2026, OpenAI and Anthropic have absorbed tool calling, streaming, function schemas, and multi-turn memory into their native SDKs. The framework’s abstractions no longer hide meaningful differences. They hide clarity. Raw API is consistently the fastest option https://www.aibuilderclub.com/blog/langchain-vs-crewai-vs-raw-api , with no framework overhead and no extra LLM calls for orchestration. Frameworks add 100–500ms of Python overhead per agent step. For latency-sensitive workloads — real-time customer support, voice agents, and high-throughput pipelines — that overhead is real and worth avoiding. Here is a complete tool-using agent built on the raw OpenAI SDK in under 80 lines. Prerequisites: pip install openai python-dotenv 1 pip install openai python-dotenv How to run: Save as raw api agent.py and run python raw api agent.py raw api agent.py A complete tool-using agent on the raw OpenAI SDK -- no framework. This is ~75 lines including comments. Compare it to the LangChain equivalent. Prerequisites: pip install openai python-dotenv How to run: python raw api agent.py import os import json from dotenv import load dotenv from openai import OpenAI load dotenv client = OpenAI api key=os.getenv "OPENAI API KEY" ── TOOL DEFINITIONS ────────────────────────────────────────────────────────── The model reads these descriptions to decide when and how to call each tool. Clear, specific descriptions are more important here than in any framework -- there is no wrapper to fill in gaps. TOOLS = { "type": "function", "function": { "name": "calculate", "description": "Evaluate a mathematical expression. Use for arithmetic, " "percentages, or numerical computation. " "Input: a Python math expression as a string." , "parameters": { "type": "object", "properties": { "expression": { "type": "string", "description": "A Python math expression, e.g. '1500 0.08'" } }, "required": "expression" } } }, { "type": "function", "function": { "name": "get word count", "description": "Count the number of words in a given string of text.", "parameters": { "type": "object", "properties": { "text": {"type": "string", "description": "The text to count."} }, "required": "text" } } } ── TOOL IMPLEMENTATIONS ────────────────────────────────────────────────────── def calculate expression: str - str: try: result = eval expression, {" builtins ": {}}, {} return str result except Exception as e: return f"Error: {e}" def get word count text: str - str: return str len text.split Maps tool name → Python function for dynamic dispatch in the loop below TOOL DISPATCH = {"calculate": calculate, "get word count": get word count} ── AGENT LOOP ──────────────────────────────────────────────────────────────── def run agent user message: str - str: """ A complete ReAct-style agent loop using raw OpenAI tool calls. The model decides whether to call a tool or return a final answer. The loop continues until the model stops requesting tool calls. Every step is visible -- no framework wrapping, no hidden logic. """ messages = {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": user message}, while True: response = client.chat.completions.create model="gpt-4o", messages=messages, tools=TOOLS, tool choice="auto", Model decides: call a tool or respond directly temperature=0, message = response.choices 0 .message messages.append message Always add the assistant message to history No tool calls = the model has its final answer if not message.tool calls: return message.content Execute each tool call the model requested for tool call in message.tool calls: name = tool call.function.name args = json.loads tool call.function.arguments fn = TOOL DISPATCH.get name result = fn args if fn else f"Unknown tool: {name}" Tool result goes back into the message history. The model reads this on the next iteration to decide what to do next. messages.append { "role": "tool", "tool call id": tool call.id, "content": result, } Loop -- the model now processes the tool results if name == " main ": queries = "What is 18% of 3500?", "How many words are in: The quick brown fox jumps over the lazy dog?", "Split 240 items into groups of 16. How many groups?", for q in queries: print f"Q: {q}\nA: {run agent q }\n" 123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123 raw api agent.py A complete tool-using agent on the raw OpenAI SDK -- no framework. This is ~75 lines including comments. Compare it to the LangChain equivalent. Prerequisites: pip install openai python-dotenv How to run: python raw api agent.py import osimport jsonfrom dotenv import load dotenvfrom openai import OpenAI load dotenv client = OpenAI api key=os.getenv "OPENAI API KEY" ── TOOL DEFINITIONS ────────────────────────────────────────────────────────── The model reads these descriptions to decide when and how to call each tool. Clear, specific descriptions are more important here than in any framework -- there is no wrapper to fill in gaps.TOOLS = { "type": "function", "function": { "name": "calculate", "description": "Evaluate a mathematical expression. Use for arithmetic, " "percentages, or numerical computation. " "Input: a Python math expression as a string." , "parameters": { "type": "object", "properties": { "expression": { "type": "string", "description": "A Python math expression, e.g. '1500 0.08'" } }, "required": "expression" } } }, { "type": "function", "function": { "name": "get word count", "description": "Count the number of words in a given string of text.", "parameters": { "type": "object", "properties": { "text": {"type": "string", "description": "The text to count."} }, "required": "text" } } } ── TOOL IMPLEMENTATIONS ──────────────────────────────────────────────────────def calculate expression: str - str: try: result = eval expression, {" builtins ": {}}, {} return str result except Exception as e: return f"Error: {e}" def get word count text: str - str: return str len text.split Maps tool name → Python function for dynamic dispatch in the loop belowTOOL DISPATCH = {"calculate": calculate, "get word count": get word count} ── AGENT LOOP ────────────────────────────────────────────────────────────────def run agent user message: str - str: """ A complete ReAct-style agent loop using raw OpenAI tool calls. The model decides whether to call a tool or return a final answer. The loop continues until the model stops requesting tool calls. Every step is visible -- no framework wrapping, no hidden logic. """ messages = {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": user message}, while True: response = client.chat.completions.create model="gpt-4o", messages=messages, tools=TOOLS, tool choice="auto", Model decides: call a tool or respond directly temperature=0, message = response.choices 0 .message messages.append message Always add the assistant message to history No tool calls = the model has its final answer if not message.tool calls: return message.content Execute each tool call the model requested for tool call in message.tool calls: name = tool call.function.name args = json.loads tool call.function.arguments fn = TOOL DISPATCH.get name result = fn args if fn else f"Unknown tool: {name}" Tool result goes back into the message history. The model reads this on the next iteration to decide what to do next. messages.append { "role": "tool", "tool call id": tool call.id, "content": result, } Loop -- the model now processes the tool results if name == " main ": queries = "What is 18% of 3500?", "How many words are in: The quick brown fox jumps over the lazy dog?", "Split 240 items into groups of 16. How many groups?", for q in queries: print f"Q: {q}\nA: {run agent q }\n" What this does: The agent loop is fully transparent. There is no framework between you and the model’s response. The while True loop runs until message.tool calls is empty, which happens when the model decides it has enough information to answer directly. Every message — system, user, assistant, and tool result — is in a plain Python list you can inspect, log, or modify at any point. That transparency is the raw path’s core advantage: when something breaks, you know exactly where to look. Head-to-Head Comparison The same task was evaluated across three dimensions. All measurements reflect current benchmarks from independent analysis cited throughout this article. Framework Overhead and Performance Metric | Raw API | LlamaIndex | LangChain LCEL | LangGraph | |---|---|---|---|---| | Framework overhead | ~0ms | ~6ms | ~10ms | ~14ms | | Token overhead per query | 0 | ~1.6K | ~2.4K | ~2.0K | | Per tool call latency | 2–5ms | N/A | 10–30ms | 10–30ms | | Stack trace depth on error | 2–5 frames | 5–10 frames | 15–40 frames | 15–40 frames | | Debug transparency | High | Medium | Low | Low | Code Volume: Same RAG Task, Three Ways This is the most concrete way to feel the trade-off. All three implementations below answer the same question from the same context document: Implementation | Lines of code | Framework install size | Debugging clarity | |---|---|---|---| | Raw OpenAI SDK | ~20 lines | openai only | Full visibility | | LlamaIndex | ~15 lines | llama-index + plugins | Medium | | LangChain LCEL | ~18 lines | langchain + langchain-openai | Low–medium | For a basic one-document Q&A, the difference is marginal. Where LlamaIndex’s code advantage compounds is when you add chunking strategies, multiple documents, re-ranking, metadata filtering, and hybrid search — each of which requires more assembly in LangChain than in LlamaIndex. When Each One Breaks Knowing when each approach fails is as useful as knowing when it succeeds. Failure mode | Raw API | LlamaIndex | LangChain | |---|---|---|---| | Retrieval accuracy degrades | You built it, you fix it | Tune chunking/index strategy | Tune each pipeline component separately | | Agent loops indefinitely | Add max iterations manually | Workflow timeout | max iterations parameter | | Prompt changes break output | Immediate, obvious | Immediate, obvious | May propagate through chain silently | | Model API changes | Update SDK | Update llama-index package | Update langchain-openai + retest | | Debugging a production error | Direct, small stack | Moderate | Deep stack traces, hard to isolate | | Scaling to high throughput | Optimal | Good | Framework overhead compounds | Full Working Example The same document Q&A task implemented three ways. Same input document, same question, different path through the stack. Read these side by side and the trade-offs become concrete. Prerequisites: pip install openai langchain langchain-openai llama-index \ llama-index-llms-openai llama-index-embeddings-openai python-dotenv 12 pip install openai langchain langchain-openai llama-index \ llama-index-llms-openai llama-index-embeddings-openai python-dotenv How to run: Save as three ways.py and run python three ways.py three ways.py The same document Q&A task implemented three ways: Raw OpenAI SDK, LlamaIndex, and LangChain LCEL. Same input. Same output. Different path through the stack. Prerequisites: pip install openai langchain langchain-openai llama-index llama-index-llms-openai llama-index-embeddings-openai python-dotenv How to run: python three ways.py import os import time from dotenv import load dotenv load dotenv QUESTION = "What is retrieval-augmented generation and why does it matter?" CONTEXT DOC = """ Retrieval-Augmented Generation RAG is a technique that improves LLM responses by fetching relevant context from an external knowledge base before generating an answer. Instead of relying solely on training data, RAG retrieves the most relevant document chunks and includes them in the prompt. This reduces hallucinations, keeps answers grounded in your actual data, and allows the model to answer questions about information it was never trained on. """ ───────────────────────────────────────────────────────────────────────────── APPROACH 1: RAW OPENAI SDK When to use: simple, one-off calls where full visibility matters most ───────────────────────────────────────────────────────────────────────────── def raw api answer question: str, context: str - str: """Answer a question using context, via raw OpenAI SDK -- no framework.""" from openai import OpenAI client = OpenAI api key=os.getenv "OPENAI API KEY" Everything is explicit: the system prompt, the context injection, the message structure. Nothing is hidden in a framework abstraction. response = client.chat.completions.create model="gpt-4o", temperature=0, messages= { "role": "system", "content": "Answer questions using only the provided context. " "If the answer is not in the context, say so clearly." }, { "role": "user", "content": f"Context:\n{context}\n\nQuestion: {question}" } return response.choices 0 .message.content ───────────────────────────────────────────────────────────────────────────── APPROACH 2: LLAMAINDEX When to use: document-heavy retrieval where you want optimized RAG out of the box ───────────────────────────────────────────────────────────────────────────── def llamaindex answer question: str, context: str - str: """Answer a question using LlamaIndex -- purpose-built retrieval pipeline.""" from llama index.core import VectorStoreIndex, Document, Settings from llama index.llms.openai import OpenAI as LlamaOpenAI from llama index.embeddings.openai import OpenAIEmbedding Configure once -- all pipeline components pick it up Settings.llm = LlamaOpenAI model="gpt-4o", temperature=0, api key=os.getenv "OPENAI API KEY" Settings.embed model = OpenAIEmbedding model="text-embedding-3-small", api key=os.getenv "OPENAI API KEY" from documents = chunk + embed + index in one call For multiple documents, pass a list: from documents doc1, doc2, doc3 index = VectorStoreIndex.from documents Document text=context as query engine = retriever + generator, wired together automatically query engine = index.as query engine similarity top k=1 return str query engine.query question ───────────────────────────────────────────────────────────────────────────── APPROACH 3: LANGCHAIN LCEL When to use: workflows that will grow to include agents, memory, or routing ───────────────────────────────────────────────────────────────────────────── def langchain answer question: str, context: str - str: """Answer a question using a LangChain LCEL chain.""" from langchain core.prompts import ChatPromptTemplate from langchain core.output parsers import StrOutputParser from langchain openai import ChatOpenAI llm = ChatOpenAI model="gpt-4o", temperature=0, api key=os.getenv "OPENAI API KEY" prompt = ChatPromptTemplate.from messages "system", "Answer using only the provided context. " "If the answer is not in the context, say so.\n\nContext:\n{context}" , "human", "{question}" The same chain supports .stream , .batch , .ainvoke -- no code changes needed chain = prompt | llm | StrOutputParser return chain.invoke {"context": context, "question": question} ───────────────────────────────────────────────────────────────────────────── RUN ALL THREE AND COMPARE ───────────────────────────────────────────────────────────────────────────── if name == " main ": approaches = "Raw OpenAI SDK", raw api answer , "LlamaIndex", llamaindex answer , "LangChain LCEL", langchain answer , for name, fn in approaches: print f"\n{'=' 60}" print f"Approach: {name}" print f"{'=' 60}" start = time.perf counter answer = fn QUESTION, CONTEXT DOC elapsed = time.perf counter - start print f"Answer: {answer}" print f"Time excluding LLM : visible in wall clock" 123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132 three ways.py The same document Q&A task implemented three ways: Raw OpenAI SDK, LlamaIndex, and LangChain LCEL. Same input. Same output. Different path through the stack. Prerequisites: pip install openai langchain langchain-openai llama-index llama-index-llms-openai llama-index-embeddings-openai python-dotenv How to run: python three ways.py import osimport timefrom dotenv import load dotenv load dotenv QUESTION = "What is retrieval-augmented generation and why does it matter?" CONTEXT DOC = """Retrieval-Augmented Generation RAG is a technique that improves LLM responsesby fetching relevant context from an external knowledge base before generatingan answer. Instead of relying solely on training data, RAG retrieves the mostrelevant document chunks and includes them in the prompt. This reduceshallucinations, keeps answers grounded in your actual data, and allows the modelto answer questions about information it was never trained on.""" ───────────────────────────────────────────────────────────────────────────── APPROACH 1: RAW OPENAI SDK When to use: simple, one-off calls where full visibility matters most ─────────────────────────────────────────────────────────────────────────────def raw api answer question: str, context: str - str: """Answer a question using context, via raw OpenAI SDK -- no framework.""" from openai import OpenAI client = OpenAI api key=os.getenv "OPENAI API KEY" Everything is explicit: the system prompt, the context injection, the message structure. Nothing is hidden in a framework abstraction. response = client.chat.completions.create model="gpt-4o", temperature=0, messages= { "role": "system", "content": "Answer questions using only the provided context. " "If the answer is not in the context, say so clearly." }, { "role": "user", "content": f"Context:\n{context}\n\nQuestion: {question}" } return response.choices 0 .message.content ───────────────────────────────────────────────────────────────────────────── APPROACH 2: LLAMAINDEX When to use: document-heavy retrieval where you want optimized RAG out of the box ─────────────────────────────────────────────────────────────────────────────def llamaindex answer question: str, context: str - str: """Answer a question using LlamaIndex -- purpose-built retrieval pipeline.""" from llama index.core import VectorStoreIndex, Document, Settings from llama index.llms.openai import OpenAI as LlamaOpenAI from llama index.embeddings.openai import OpenAIEmbedding Configure once -- all pipeline components pick it up Settings.llm = LlamaOpenAI model="gpt-4o", temperature=0, api key=os.getenv "OPENAI API KEY" Settings.embed model = OpenAIEmbedding model="text-embedding-3-small", api key=os.getenv "OPENAI API KEY" from documents = chunk + embed + index in one call For multiple documents, pass a list: from documents doc1, doc2, doc3 index = VectorStoreIndex.from documents Document text=context as query engine = retriever + generator, wired together automatically query engine = index.as query engine similarity top k=1 return str query engine.query question ───────────────────────────────────────────────────────────────────────────── APPROACH 3: LANGCHAIN LCEL When to use: workflows that will grow to include agents, memory, or routing ─────────────────────────────────────────────────────────────────────────────def langchain answer question: str, context: str - str: """Answer a question using a LangChain LCEL chain.""" from langchain core.prompts import ChatPromptTemplate from langchain core.output parsers import StrOutputParser from langchain openai import ChatOpenAI llm = ChatOpenAI model="gpt-4o", temperature=0, api key=os.getenv "OPENAI API KEY" prompt = ChatPromptTemplate.from messages "system", "Answer using only the provided context. " "If the answer is not in the context, say so.\n\nContext:\n{context}" , "human", "{question}" The same chain supports .stream , .batch , .ainvoke -- no code changes needed chain = prompt | llm | StrOutputParser return chain.invoke {"context": context, "question": question} ───────────────────────────────────────────────────────────────────────────── RUN ALL THREE AND COMPARE ─────────────────────────────────────────────────────────────────────────────if name == " main ": approaches = "Raw OpenAI SDK", raw api answer , "LlamaIndex", llamaindex answer , "LangChain LCEL", langchain answer , for name, fn in approaches: print f"\n{'=' 60}" print f"Approach: {name}" print f"{'=' 60}" start = time.perf counter answer = fn QUESTION, CONTEXT DOC elapsed = time.perf counter - start print f"Answer: {answer}" print f"Time excluding LLM : visible in wall clock" What this does: All three functions receive the same QUESTION and CONTEXT DOC and return a string answer. The raw API version manually constructs the message list and extracts the response. The LlamaIndex version uses from documents and as query engine to handle the pipeline. The LangChain version assembles a prompt, model, and parser with the | operator. At this scale — one document, one question — the differences are minimal. Feed this function 500 documents and a complex query, and the gap between LlamaIndex’s purpose-built retrieval and the other two approaches opens up significantly. Wrapping Up The framework decision is not about which option has the most GitHub stars or the most features. It is about matching the abstraction level of your tool to the actual complexity of your problem. For simple, one-shot tasks, raw API calls are faster to write, faster to run, and easier to debug than any framework. For document retrieval at any meaningful scale, LlamaIndex earns its dependency through better chunking, faster indexing, and less code. For stateful agents with memory, tools, and multi-step reasoning, LangGraph’s persistence and graph-based control flow are genuinely hard to replicate cleanly with a hand-rolled loop. The pattern that most production teams converge on by mid-2026 https://callsphere.ai/blog/llm-orchestration-langchain-llamaindex-comparison is not a single framework but a layered stack: raw SDK for the simple calls, LlamaIndex for the retrieval layer, LangGraph for the agent loop, and LangSmith for tracing across everything. None of those choices locks you out of the others. They compose. The practical rule is this: start with the minimal option that handles your current requirements, and add a framework when you hit a problem the framework was built to solve — not before. A retrieval problem you encounter is a reason to add LlamaIndex. A state management problem you encounter is a reason to add LangGraph. Adding either before you feel the pain they address means adding maintenance overhead for a future problem that may not arrive in the shape you expected.