{"slug": "stop-stalking-your-crush-stalk-your-agents-instead-a-langsmith-deep-dive-part-1", "title": "Stop Stalking Your Crush, Stalk Your Agents Instead: A LangSmith Deep Dive: Part -1", "summary": "LangSmith, a monitoring and observability platform built by the creators of LangChain and LangGraph, traces AI applications by logging every input and output across each step of a pipeline, according to a technical deep dive on the tool. The article distinguishes observability, which shows what happened and why, from monitoring, which tracks overall metrics such as latency and the cost of one end-to-end execution. It notes that a multi-step LLM pipeline may make 5 to 10 LLM calls per user request, and that non-deterministic outputs mean a failed run cannot be reproduced unless it was logged.", "body_md": "**LangSmith** is a monitoring and observability platform built by the creators of LangChain and LangGraph for tracing AI applications.\n\nBut before diving into LangSmith, let’s first understand what observability and monitoring actually mean.\n\n**Observability** is essentially keeping a close eye on your AI applications while they run — tracking exactly what input goes into each step and what output comes out of it. Take a RAG (Retrieval-Augmented Generation) application as an example. It’s made up of several moving parts: vector stores, retrievers, documents, embeddings, and the LLM itself. Observability here means logging every single input and output as it flows between these components. This way, when something breaks (and something always breaks), you know exactly where to look instead of guessing.\n\n**Monitoring**, although it sounds similar to observability, is actually a different concept. Monitoring is the process of tracking your system or application’s metrics as a whole — things like latency across different runs, the cost of one end-to-end execution, and so on.\n\n*In short*: Observability tells you what happened and why, while monitoring tells you how well things are performing overall. You need both — monitoring flags that something’s wrong (say, latency spiked at 3 PM), while observability helps you drill down and find out exactly which component caused it.\n\nWhy do LLM apps specifically need this?\n\nTraditional software is predictable. If you call a function with the same input, you get the same output — every single time. When something breaks, you add a print statement, check the logs, find the line, fix it. Done.\n\nLLM applications don’t work like that.The output is non-deterministic. The same prompt can produce a different response on every single run. So when a user complains “the answer was wrong,” you can’t just reproduce it and debug it. That exact run is gone — unless you logged it.\n\nThe pipeline has multiple steps, each of which can silently fail. Take a RAG application. When your app gives a wrong answer, where did it go wrong?\n\n- Did the vector store retrieve the wrong documents?\n\n- Did the retriever rank them poorly?\n\n- Did the prompt template stuff too much context in?\n\n- Did the LLM just hallucinate despite having the right context?\n\n- Or did the change in prompt caused it?\n\nWithout observability you can just guess and play catch up but not get the exact root cause of the problem. You’d have to manually test each component in isolation, which is slow and doesn’t reflect what actually happened during that specific run. And if the workflow or application is complex containing lot of components, finding the issue would become a nightmare.\n\nEvery LLM call costs money — and in a multi-step pipeline, you might be making 5 to 10 LLM calls per user request without realizing it. Without monitoring, you have no idea which step is burning your budget. Is it the query rewriter? The summarizer? The final answer generator? You won’t know until your API bill arrives.\n\n*This is exactly why LangSmith exists.*\n\n**Let’s learn about some core concepts of LangSmith -**\n\n1. Projects — A Project in LangSmith is simply a container for one of your AI applications. Every trace and run gets logged under a project so your data stays organized and separated.\n\n2. Trace — A Trace represents one complete end-to-end execution of your application — from the moment a user sends an input to the moment your app returns a final response.\n\n> For example, a user asks: “What is the return policy?”\n\n> That single question triggers your entire RAG pipeline — retrieval, reranking, prompt construction, LLM call, response generation. All of that together, from start to finish, is one trace.\n\n3. Runs — If a Trace is the full journey, a run is each step along the way.\n\nInside that one trace of “What is the return policy?”, LangSmith breaks it down into runs:\n\nTrace: “What is the return policy?”\n\n- Run 1: Embed the user query \n\n- Run 2: Retrieve documents \n\n- Run 3: Rerank documents \n\n- Run 4: Construct prompt \n\n- Run 5: LLM call\n\n**Setting up LangSmith**\n\nStep 1: Create a LangSmith Account\n\nHead over to smith.langchain.com and sign up. Once you’re in, navigate to Settings → API Keys and generate a new API key. Copy it somewhere safe.\n\nStep 2: Create a Project\n\nOnce you’re inside the LangSmith dashboard, create a new project. Give it a meaningful name that matches your application.\n\nStep 3: Install the Package\n\n```\npip install langsmith\n```\n\nStep 4: Make sure to have these environment variables\n\n```\n.envLANGCHAIN_TRACING_V2=true - Turns tracing on/off - set to true to enableLANGCHAIN_API_KEY=your-langsmith-api-keyLANGCHAIN_PROJECT=your-project-name - Which project to send traces toThat’s literally it. These three environment variables are all LangSmith needs to start capturing traces.\n```\n\n**Let’s see a simple langchain workflow in action tracked by LangSmith -**\n\n``` python\nfrom dotenv import load_dotenvfrom langchain_openai import ChatOpenAIfrom langchain_core.prompts import PromptTemplatefrom langchain_core.output_parsers import StrOutputParser# Load environment variablesload_dotenv()# Prompt to generate a detailed reportprompt1 = PromptTemplate( template=\"Generate a detailed report on {topic}\", input_variables=[\"topic\"],)# Prompt to summarize the reportprompt2 = PromptTemplate( template=\"Generate a 5 pointer summary from the following text:\\n\\n{text}\", input_variables=[\"text\"],)# Initialize modelsmodel1 = ChatOpenAI(model=\"gpt-4o-mini\")model2 = ChatOpenAI(model=\"gpt-4o\")# Output parserparser = StrOutputParser()# Create sequential chainchain = ( prompt1| model1| parser| prompt2| model2| parser)# Configuration for tracingconfig = { \"tags\": [\"llm_app\", \"report generation\", \"summarization\"]}# Invoke chainresult = chain.invoke( {\"topic\": \"Artificial Intelligence\"}, config=config,)print(result)\n```\n\n![LangSmith Dashboard]([https://dev-to-uploads.s3.us-east-2.amazonaws.com/uploads/articles/6gch9fyouc4ugzz8i8px.png](https://dev-to-uploads.s3.us-east-2.amazonaws.com/uploads/articles/6gch9fyouc4ugzz8i8px.png))\n\nThe screenshot above shows a real LangSmith trace for a simple Sequential LLM application.\n\nThe middle panel breaks down the six runs inside this trace — `PromptTemplate` formatted the input, gpt-4o-mini made the first LLM call (13.68s, 1.1K tokens), `StrOutputParser` cleaned the output, then the chain continued with another `PromptTemplate`, a second LLM call via gpt-4o (4.85s, 1.4K tokens), and a final `StrOutputParser`. On the right, you can see the exact input (topic: AI Opportunity in India) and the structured output the LLM returned.\n\nThis is exactly what makes LangSmith powerful. When something goes wrong, you don’t guess — you just open the trace and see precisely where it broke.\n\nLangSmith works out of the box with LangChain and LangGraph — no extra setup needed. However, if your pipeline includes components that aren’t natively part of these frameworks, LangSmith won’t trace them automatically. For those cases, you can wrap the function with the *@traceable* decorator and LangSmith will capture it just like any other run.\n\n``` python\nfrom langsmith import traceablefrom openai import OpenAIclient = OpenAI()@traceable # LangSmith will trace this functiondef call_llm(question: str) -> str: response = client.chat.completions.create( model=\"gpt-4o-mini\", messages=[{\"role\": \"user\", \"content\": question}] )return response.choices[0].message.contentresult = call_llm(\"What is RAG in AI?\")print(result)The `@traceable` decorator tells LangSmith — “treat this function as a run, log its input and output.” Works with any Python function, any framework.\n```\n\n**Alternatives of LangSmith**\n\n**Langfuse**\n\nLangfuse is open source and self-hostable, meaning your data never leaves your own infrastructure. It works with any LLM framework — not just LangChain — and comes with prompt versioning and evaluation built in. Best choice if data privacy is a concern.\n\n**Helicone**\n\nHelicone works as a proxy between your app and the LLM provider. You change one base URL and it automatically starts logging every request — tokens, cost, latency. No code changes needed. Best for teams who just want clean cost and usage visibility without a full observability setup.\n\n**Arize Phoenix**\n\nPhoenix runs completely locally — no cloud, no data sharing. It goes beyond just tracing, offering embeddings visualization and dataset analysis. Best suited for ML teams doing serious evaluation or fine-tuning work alongside production monitoring.\n\n**Conclusion**\n\nLet’s zoom out and look at what we covered -\n\nWe started with a simple truth — LLM applications are fundamentally different from traditional software. They’re non-deterministic, multi-step, and fail silently. A wrong answer doesn’t throw an error. It just quietly erodes your user’s trust until they stop using your product.\n\nObservability and monitoring are your defense against that. Observability tells you what happened and why at every step. Monitoring tells you how well your system is performing over time. You need both. LangSmith gives you both — wrapped in a clean UI that integrates natively with LangChain and LangGraph with almost zero setup effort.\n\nThe best time to add observability to your LLM app is before you need it — because by the time something breaks in production and a user is complaining, you’ll wish you had the trace from that exact run.\n\nSo stop stalking your crush and start stalking your agents. Every trace tells a story, and the better you understand those stories, the easier it becomes to build reliable AI applications.\n\nIn Part 2, we’ll go beyond tracing and explore how LangSmith helps you evaluate prompts, create datasets, run experiments, collect user feedback, and continuously improve your LLM applications.\n\n[Stop Stalking Your Crush, Stalk Your Agents Instead: A LangSmith Deep Dive: Part -1](https://pub.towardsai.net/stop-stalking-your-crush-stalk-your-agents-instead-a-langsmith-deep-dive-part-1-924c37fb8364) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.", "url": "https://wpnews.pro/news/stop-stalking-your-crush-stalk-your-agents-instead-a-langsmith-deep-dive-part-1", "canonical_source": "https://pub.towardsai.net/stop-stalking-your-crush-stalk-your-agents-instead-a-langsmith-deep-dive-part-1-924c37fb8364?source=rss----98111c9905da---4", "published_at": "2026-09-15 04:00:17+00:00", "updated_at": "2026-09-15 04:34:09.490601+00:00", "lang": "en", "topics": ["ai-tools", "large-language-models", "mlops", "developer-tools", "ai-products"], "entities": ["LangSmith", "LangChain", "LangGraph"], "alternates": {"html": "https://wpnews.pro/news/stop-stalking-your-crush-stalk-your-agents-instead-a-langsmith-deep-dive-part-1", "markdown": "https://wpnews.pro/news/stop-stalking-your-crush-stalk-your-agents-instead-a-langsmith-deep-dive-part-1.md", "text": "https://wpnews.pro/news/stop-stalking-your-crush-stalk-your-agents-instead-a-langsmith-deep-dive-part-1.txt", "jsonld": "https://wpnews.pro/news/stop-stalking-your-crush-stalk-your-agents-instead-a-langsmith-deep-dive-part-1.jsonld"}}