If you need a Python library to build an AI agent that can run in production, start with LangChain for flexibility, CrewAI for team-style orchestration, or LlamaIndex if your focus is on data-centric retrieval. All three install with a single pip
command, but they differ in abstraction level, runtime cost, and how easy they are to test inside a FastAPI service.
The three most battle-tested options today are LangChain, CrewAI, and LlamaIndex.
All three are open source, support OpenAI, Anthropic, and Azure endpoints, and have async APIs that play nicely with FastAPI.
Installation is straightforward, but each library pulls a different set of optional dependencies.
pip install langchain crewai llama-index
pip install openai anthropic
pip install chromadb
python
from langchain.llms import OpenAI
from langchain.prompts import PromptTemplate
from langchain.chains import LLMChain
llm = OpenAI(model="gpt-4o-mini")
prompt = PromptTemplate.from_template("Translate this to French: {text}")
chain = LLMChain(llm=llm, prompt=prompt)
python
from crewai import Agent, Task, Crew
researcher = Agent(
role="Researcher",
goal="Find the latest Python web frameworks",
backstory="You are a senior backend engineer.",
llm="gpt-4o-mini",
)
task = Task(
description="List three frameworks with a one‑sentence summary each.",
agent=researcher,
)
crew = Crew(agents=[researcher], tasks=[task])
python
from llama_index import VectorStoreIndex, SimpleDirectoryReader
from llama_index.llms import OpenAI
documents = SimpleDirectoryReader("data/").load_data()
index = VectorStoreIndex.from_documents(documents, llm=OpenAI(model="gpt-4o-mini"))
All three snippets run in a fresh virtual environment. The biggest gotcha I’ve hit is version mismatches between langchain
and crewai
when they both depend on openai
. Pinning openai>=1.0,<2.0
in requirements.txt
saved me a day of debugging.
Below is a tiny “weather lookup” agent that takes a city name, calls a mock weather API, and returns a friendly sentence. The logic is the same; only the orchestration differs.
import httpx
from langchain.llms import OpenAI
from langchain.prompts import PromptTemplate
from langchain.chains import LLMChain
llm = OpenAI(model="gpt-4o-mini")
prompt = PromptTemplate.from_template(
"You are a helpful assistant. Use the JSON response {weather} to answer: "
"What is the weather like in {city}?"
)
async def fetch_weather(city: str) -> dict:
async with httpx.AsyncClient() as client:
resp = await client.get(f"https://api.example.com/weather?city={city}")
return resp.json()
async def weather_agent(city: str) -> str:
weather = await fetch_weather(city)
chain = LLMChain(llm=llm, prompt=prompt)
return await chain.arun(weather=weather, city=city)
python
from crewai import Agent, Task, Crew
import httpx
weather_agent = Agent(
role="Weather Bot",
goal="Provide a short weather summary",
backstory="You are a backend engineer who loves concise messages.",
llm="gpt-4o-mini",
)
async def fetch_weather(city):
async with httpx.AsyncClient() as client:
r = await client.get(f"https://api.example.com/weather?city={city}")
return r.json()
task = Task(
description="Summarize the JSON weather data for {city}",
expected_output="A one‑sentence weather report.",
agent=weather_agent,
async_func=fetch_weather, # CrewAI lets you attach a callable
)
crew = Crew(agents=[weather_agent], tasks=[task])
async def run_crew(city):
return await crew.kickoff(inputs={"city": city})
python
from llama_index import SimpleDirectoryReader, VectorStoreIndex, ServiceContext
from llama_index.llms import OpenAI
import httpx
documents = SimpleDirectoryReader(input_files=["prompt.txt"]).load_data()
service_context = ServiceContext.from_defaults(llm=OpenAI(model="gpt-4o-mini"))
index = VectorStoreIndex.from_documents(documents, service_context=service_context)
async def fetch_weather(city):
async with httpx.AsyncClient() as client:
r = await client.get(f"https://api.example.com/weather?city={city}")
return r.json()
async def llama_agent(city):
query = f"Summarize this JSON: {await fetch_weather(city)}"
response = await index.as_query_engine().aquery(query)
return response.response
All three agents return a string like “In Paris it’s 12 °C with light rain.” The LangChain version feels most explicit; CrewAI hides the prompt plumbing but requires you to think in terms of tasks; LlamaIndex bundles the prompt inside an index, which can be overkill for a single API call.
| Feature | LangChain | CrewAI | LlamaIndex |
|---|---|---|---|
| Prompt control | |||
| Full, low-level | Medium (templates per task) | Low (index-driven) | |
| Tool integration | |||
Easy via Tool class |
|||
| Built-in task-to-tool mapping | Requires custom retriever | ||
| Memory | |||
Built-in ConversationBufferMemory |
|||
| Implicit via task history | Index acts as persistent memory | ||
| Async support | |||
| Native async methods | Async tasks supported | Async query engine | |
| Cost predictability | |||
| You manage each LLM call | Crew may fire extra calls for coordination | Index creation can add one-off token cost | |
| Learning curve | |||
| Steeper, many concepts | Gentle, opinionated | Gentle if you only need retrieval |
In my production services, LangChain adds ~15 ms per extra tool call, CrewAI adds ~30 ms overhead for task orchestration, and LlamaIndex adds a one-time indexing cost that can be several seconds for a 10 k document corpus. If you’re budget-conscious, watch out for CrewAI’s hidden “role-play” calls – they can double your token bill if you forget to set max_tokens
on the underlying LLM.
FastAPI is already async-first, so you can drop any of the async agents directly into a route. Below is a minimal FastAPI app that uses the LangChain weather agent; swap the import for crew_agent
or llama_agent
to try the other libraries.
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
import uvicorn
from agents.langchain_weather import weather_agent # or crew_agent, llama_agent
app = FastAPI(title="Weather AI Agent")
class CityRequest(BaseModel):
city: str
@app.post("/weather")
async def get_weather(req: CityRequest):
try:
result = await weather_agent(req.city)
return {"city": req.city, "report": result}
except Exception as exc:
raise HTTPException(status_code=502, detail=str(exc))
if __name__ == "__main__":
uvicorn.run("app.main:app", host="0.0.0.0", port=8000, reload=True)
httpx
timeouts (client = httpx.AsyncClient(timeout=10.0)
).
OpenAI()
inside the route handler.If you need more context, see my earlier post on AI agent python ollama: Build, Test, Deploy with FastAPI for a full deployment pipeline.
Testing LLM-driven code feels weird because the output is nondeterministic. Here’s what I rely on:
assert "Translate this to French" in prompt
. If the prompt changes, the test fails before you even call the model.
OpenAI
with a stub that returns a deterministic JSON payload. Libraries like pytest-mock
let you patch OpenAI.__call__
. This isolates the business logic from the provider.
httpx.AsyncClient(app=app)
in pytest to hit /weather
with a fake city and assert the shape of the JSON response.
max_tokens
parameter let the model keep streaming.When you’re ready to scale, read the “Testing and validation” chapter of the official LangChain docs – they outline a “LLMMock” helper that integrates nicely with pytest.
Which library should I start with for a simple chatbot?
LangChain gives the most flexibility and the smallest dependency footprint for a single-turn bot. If you want built-in task management, try CrewAI.
Do I need a vector store for every agent?
Only if you plan to retrieve or embed large text collections. LlamaIndex shines there; otherwise it adds unnecessary latency.
Can I swap OpenAI for an on-prem model?
All three libraries expose a generic LLM interface. Point the client to your local server (e.g., OpenAI(api_base="http://localhost:8000/v1")
) and the rest of the code stays the same.
How do I monitor token usage per request?
Wrap the LLM call in a decorator that reads the usage
field from the response and pushes it to Prometheus or CloudWatch. I keep a request_id
in the FastAPI state
to correlate logs.
pip install
line; watch version pins for openai
.