LangSmith for Monitoring Non-Deterministic Agent Workflows

LangChain's LangSmith observability platform can record every step of non-deterministic AI agent workflows by organizing data into OpenTelemetry-like runs and trace trees, according to a technical walkthrough published by LangChain. The guide instructs developers to install the langsmith and openai packages, set LANGSMITH_TRACING="true" along with LANGSMITH_API_KEY and LANGSMITH_PROJECT, and wrap the OpenAI client with wrap_openai to capture traces, feedback, and regression datasets. The article notes that a region mismatch between LANGSMITH_ENDPOINT and the API key is the most common cause of authentication errors, such as a 401 on the first run.

An agent loop, a set of tools, integration with MCP, and a reasoning model make every run of your AI agent unique. We all know that this behavior is non-deterministic . It creates a specific operations problem. You cannot reproduce a failure from the input alone. You need a record of every step. This article shows how to build that record with LangSmith https://docs.langchain.com/langsmith/observability , LangChain’s observability and evaluation platform. You will build a small agent, instrument it step by step, run it live, and finish with traces, feedback, and a regression dataset in your LangSmith workspace. A single LLM call is easy to debug. You send a prompt, you get a response. An agent is different. An agent runs a loop of Think → Act → Observe . The model picks a tool. The harness e.g. Claude Code runs it. The model sees the result and picks the next step. Three problems follow from this design. The agent can pick the wrong tool or loop on a failing step while the final output still looks correct. The agent decides its own cost, so one run makes three tool calls and the next can make ten. Small per-step errors compound over long tasks. Traditional logging does not show structure, cost, or timing. You need structured traces. LangSmith organizes data into a few core terms. The definitions below come from the LangSmith observability concepts https://docs.langchain.com/langsmith/observability-concepts documentation. The terminology is deliberately OpenTelemetry-like. Runs are spans, traces are trace trees, and LangSmith accepts standard OpenTelemetry instrumentation alongside its own SDKs. Install the packages: pip install -U langsmith openai Set the environment variables: export LANGSMITH TRACING="true"export LANGSMITH API KEY="lsv2 pt ..."export LANGSMITH PROJECT="langsmith-medium-article"export OPENROUTER API KEY="sk-or-..." a free OpenRouter key. The open-source model runs via OpenRouterRouter key LANGSMITH TRACING=true is the switch that turns recording on. LANGSMITH PROJECT sends all traces to a named project; if you omit it, LangSmith creates a default project automatically. If your account is outside the US, also set the endpoint for your region, for example LANGSMITH ENDPOINT=”https://eu.api.smith.langchain.com" for EU accounts. Do not add a trailing slash. A region mismatch is the most common cause of authentication errors on the first run. These are terminal commands, so they use the export keyword. If you prefer a .env file, write plain KEY=value lines without export and load it in your script with python-dotenv . And if your console ever shows Failed to export span batch … 401 , your LANGSMITH ENDPOINT region does not match your API key. The demo agent answers questions about internal meeting notes. It has two tools and a step limit. The only LangSmith-specific lines are the import and the client wrapper: python import jsonimport astimport osfrom openai import OpenAIfrom langsmith.wrappers import wrap openaifrom langsmith import traceablefrom langsmith import Clientclient = wrap openai OpenAI base url="https://openrouter.ai/api/v1", api key=os.environ "OPENROUTER API KEY" , the open-source model runs via OpenRouter MODEL = "nvidia/nemotron-3.5-lightning:free"This is the complete demo script. Save it as agent.py — every step that follows edits this same file, so you can run the full article from one place. Run it twice. The paths differ. That is the monitoring problem, and the rest of this article solves it. The @traceable decorator records one run per function call. run agent becomes the root run of a trace. Add run type="tool" to the tool executor above, and it becomes a nested tool run. @traceable run type="tool" every tool call becomes a nested run: arguments, result, timingdef execute tool name: str, arguments: str - str: ... Why two decorators? The decorator on run agent records the whole agent run as one trace. The decorator on execute tool records each tool call as a nested tool run. Without it, tool calls would be invisible: you would see the model announce a tool, but not what the tool did, what it returned, or how long it took. Run the demo and open the Tracing page in the LangSmith UI. Every run appears as one row, and opening a row shows the full run tree: the agent run, the tool runs nested under it, and the model calls nested under those, each with inputs, outputs, latency, and token cost. A user conversation spans many agent runs. LangSmith groups them with a thread id. Update the decorator on run agent from Step 3: @traceable metadata={"thread id": "support-agent-thread-001"} same decorator, plus the thread IDdef run agent question: str, max steps: int = 5 - str: ... Every trace that carries the same thread id appears as one thread, turn by turn. LangSmith adds a second view on top: the trajectory , a flattened list of every message exchanged in the session, in order. The thread keeps the nesting; the trajectory shows the conversation. Use the thread to debug execution, use the trajectory to read what happened. The thread stats tell the story in one glance: eight turns, feedback averaging 0.75, and a token breakdown that is 89% input — agent loops spend most of their budget re-reading context, not writing answers. A trace shows behavior. Feedback will add the judgment. Feedback is a score bound to a run by its run ID. Who gives the score? In production, usually your users: a thumbs-up button in your app, wired to the run ID of that conversation. It can also be your code running automated checks, or a reviewer scoring runs manually in the LangSmith UI. In this demo there is no product UI, so the script scores its own runs: real answers get a 1, escalations get a 0. Same mechanics as a rating button, minus the button. This block runs after your agent runs, in the same script or a separate scoring script: python from langsmith import Clientclient = Client client.create feedback run id, the ID of the run you are rating key="user feedback", 1 = helpful, 0 = not helpful score=1, Where does run id come from? The demo gets it from the project itself: list the root runs one root run per agent run , and score each one. This is the block at the end of the demo script: roots = list client.list runs project name="langsmith-medium-article", is root=True, limit=20 for r in roots: question = r.inputs.get "question", "" the question this run answered helpful = 1 if "Step limit" not in str r.outputs else 0 1 = answered, 0 = escalated client.create feedback r.id, key="user feedback", score=helpful Prefer binary scores. A score of 1 or 0 forces a precise definition of correct behavior. LangSmith stores regression tests as datasets of examples. Each example holds an input and a reference output: the container that holds all your regression tests:client.create dataset dataset name="support-agent-regression" each item pairs a question with the answer that "correct" looks like:client.create example dataset name="support-agent-regression", inputs={"question": "What was the total Q3 spend?"}, outputs={"answer": "$60,500"}, the reference output: what a correct response contains client.create example dataset name="support-agent-regression", inputs={"question": "What did we decide about the mobile redesign?"}, outputs={"answer": "Delayed to Q1"}, another reference output In the demo script, add this block at the very end, right after Client .flush : the agent runs first, the traces get sent, and then the questions become dataset items. You can also run it as its own small script, as long as the dataset and examples are created at least once. There is a shortcut for production failures: client.create example from run run id, dataset name=... converts a real failing run into a dataset item in one call. Then run experiments : LangSmith executes your target function against every example, records the outputs and scores, and displays experiments side by side so you can compare prompt versions, models, or tool sets. A change ships only when the scores hold. This is the closed loop for a non-deterministic system. You cannot replay a run exactly. You can test a change against every failure you have ever recorded. LangSmith’s evaluation framework supports three techniques, and operations teams can use all three. When several scores matter together, a composite evaluator combines them into one: a weighted average of, say, correctness and citation accuracy, attached back to the run as a single release score. Run one experiment per application version, compare them on the same dataset, and gate releases on the scores. Offline evaluation validates before deployment; online evaluation monitors live traffic after it. Note the second row: the agent’s answer was semantically correct, but my exact-match evaluator scored it 0 because the wording differed. Code evaluators check literal patterns. When wording varies, that is what LLM-as-judge evaluators are for. After deployment, you watch aggregates. LangSmith dashboards provides cost, latency, volume, and feedback over time, with alerts that fire through Slack, webhooks, or GitHub Actions when a metric leaves its range. Two agent-specific views carry the most weight: filter runs for a high tool-call count with a low distinct-tool count to find doom loops, and watch feedback trends after every release. LangSmith also ships Engine https://www.langchain.com/blog/introducing-langsmith-engine , which detects recurring failure patterns in traces automatically and proposes fixes — worth turning on once your project has real traffic. Two platform pieces sit on top of observability. LangSmith Studio https://docs.langchain.com/langsmith/studio is an agent IDE: it visualizes your graph, runs your agent interactively, manages threads and memory, and runs dataset experiments — with one-click deploy to LangSmith Cloud. LangSmith Deployment is the runtime for hosting agents LangGraph, Claude Agent SDK, CrewAI, and others with durable execution, threads, and streaming. For a monitoring-focused article it is enough to know they exist: observability, evaluation, and deployment share the same trace data. Both platforms trace non-deterministic agent workflows, and both are open source at the core. The differences from running the same agent on both: LangSmith’s threads and trajectories give the best conversation-level view, annotation queues and the experiments UI are the most mature offline-evaluation workflow, and the LangGraph integration is a single environment variable. Langfuse https://medium.com/towards-artificial-intelligence/langfuse-for-monitoring-non-deterministic-agent-workflows-a669dc7ecc1f?sharedUserId=sac.anand1 answers with fully self-hostable data regions, prompt versioning with labels, and a lighter setup path for non-LangChain stacks. Pick by ecosystem: if you build on LangChain or LangGraph, LangSmith is the native choice; if you need strict data residency or a provider-neutral harness, Langfuse fits cleanly. A non-deterministic agent will fail many times, and you must record. LangSmith gives that record a structure: runs for steps, traces for operations, threads for conversations, trajectories for the story, feedback for judgment, and datasets for regression testing. The setup in this article takes less than one hour. The payoff starts with the first production failure, when you open the run tree and see the exact step where the agent went wrong. Start with one decorator and one wrapper. Add datasets and evaluators when the first real failures arrive. LangSmith for Monitoring Non-Deterministic Agent Workflows https://pub.towardsai.net/langsmith-for-monitoring-non-deterministic-agent-workflows-741262d19fb0 was originally published in Towards AI https://pub.towardsai.net on Medium, where people are continuing the conversation by highlighting and responding to this story.