{"slug": "presentation-from-ai-agent-demo-to-production-automated-testing-and-evaluation", "title": "Presentation: From AI Agent Demo to Production: Automated Testing and Evaluation", "summary": "Zhou Yu, co-founder of Arklex AI and a professor at Columbia University, stated that 95% of AI agents remain in demos and fail to create real value, during a presentation on simulation-driven agent testing and improvement. He highlighted production examples such as Walmart's shopping agent Sparky and Amazon's Rufus, and demonstrated a voice agent for credit card recommendations.", "body_md": "## Transcript\n\n**Zhou Yu:** Today I'm going to talk about simulation-driven AI agent testing and improvement. A little bit of background. I'm the co-founder of Arklex AI. I'm also a professor at Columbia University. I've been working on AI for quite a long time. I've been a professor for nine years. I've been working with various open-source models for pre-training, post-training, and so on. One of the things I really cared about is how do these models actually get into production? How can we actually utilize these models' abilities to actually perform real things that changes people's lives? One of the things that we noticed today and we have seen in a lot of reviews is like 95% of agents are staying in demos. They're very interesting to play with, they're very nice demos to look at, but they can't really create real values, and really making changes within the process we have been experiencing for many years. I'm going to talk about a couple of applications we have been working on before, either in my lab and/or in my startup. My startup is a spinoff of my lab's research, specifically on simulations.\n\n## Conversational/Chat-Based Agents\n\nThere are so many different kinds of agents. Today I'm going to talk about one kind of agent first, which is what we usually call conversational agents or chat-based agents. It doesn't mean that it only does like input as text and output as text. There is also actions and tools involved. In general, these lines of agents are very interested in terms of like you have multiple rounds of interactions with your end users. Here we're going to talk about one of these kinds of agents we usually call consumer-facing agents. This one is called shopping agents, which Walmart has had already in production for more than a year. If you open up the Walmart app, so basically you will see this yellow dotted, like a happy smiling face. This one is called Sparky. Amazon also has a similar one called Rufus. These shopping agents have been pretty popular within all these e-commerce platforms.\n\nHere I'm going to play a little bit of one of the applications that people have been building on top of this Sparky. Here we can see when you look at a product, there are pre-generated question-answer pairs about this particular product. None of them are really what you wanted to know. You can actually ask very specific natural language questions. You can ask about some aspects of the product, for example, what's the size of the vacuum cleaner, and so on. This is what we usually call a question-answer. Of course, nowadays, we also combine it with tools like bringing the product review card into this so that you can scroll over. Then there's also what we call a contextual understanding. If you ask another question, we know this is within the context of vacuum cleaner. You can also connect to the tools of the inventory and provide recommendations.\n\nThen you can now also visualize into a product card. Then you can click on this product card and then actually do more things with it. A user can actually interact with these agents through various processes, and ask more proactive questions or answering these proactive questions from the user. We definitely see now agents are not the dumb chatbots we used to experience. They can have all these enrichment with different tools, enrichment of your user data in order to make the experience more like how real salespeople would have interacted with a real customer in a physical store. This is just one example.\n\n## Demo - Voice Agent\n\nWe also see there are a lot of what we call voice agents out there.\n\n**Customer:** Hi, I'm interested in getting a credit card, and one that fits my lifestyle.\n\n**Agent:** Absolutely, I'd be happy to help. Could you share what types of things you usually spend on, or any features you're looking for?\n\n**Customer:** I order a lot of food delivery, especially through Uber Eats. I shop online often, usually Amazon, and I occasionally book travel. I want something flexible when it comes to redeeming rewards.\n\n**Agent:** Thanks for sharing. Based on that, I'd recommend the Emerald Preferred Card. It's a great match for your habits. You'll earn three times point on Uber Eats and DoorDash. Five times points on travel booked through iChase Travel. $50 annual hotel credit. Free year of DashPass, 60,000 point welcome bonus with up to $750 when used for travel.\n\n**Customer:** Sounds good. How do I redeem the welcome points?\n\n**Agent:** Your points are fully flexible. You can redeem them for travel, cashback, statement credits, Amazon purchases, and more. Would you like to apply?\n\n**Customer:** Yes. Could you help me with the application?\n\n**Agent:** Of course. I've just sent you a secure link via SMS, follow the steps, and I'll be here if you need help.\n\n**Customer:** Amazing. Thanks so much.\n\n**Agent:** Awesome. You should hear back in three to seven business days. Enjoy your new card.\n\n**Zhou Yu:** This is the first interaction, and then now we have the second interaction.\n\n**Agent:** Just reaching out to let you know your Emerald Preferred Card has been approved. Congratulations.\n\n**Customer:** Thanks. We talked about a few benefits earlier, but I forgot the details on the travel reward. Could you remind me how that works again?\n\n**Agent:** Absolutely. You'll earn five times points on any travel booked through iChase Travel. Plus, you'll get a $50 hotel credit each year.\n\n**Customer:** Got it. What's the annual fee again?\n\n**Agent:** It's $95, but most customers easily cover that with a welcome bonus worth $750. The rewards are also flexible. You can redeem points for cash, gift cards, Amazon, or travel.\n\n**Customer:** Amazing. Ok. Can I activate the card now?\n\n**Agent:** Sure. I just sent you a secure activation link via SMS. Let me know if you need anything.\n\n**Customer:** Yes. Got it. It's activated.\n\n**Agent:** Great. If you have questions about rewards or DashPass, just text or call anytime. Enjoy the perks.\n\n**Customer:** Thanks a lot, Jordan.\n\n**Agent:** You're welcome. Have an awesome day.\n\n**Zhou Yu:** This is actually a demo. It's not actually in production. One of the biggest reasons is this is a very fancy demo. We can see there is voice and there is also text interaction. In compliance-heavy areas like finance, it's very difficult to deploy these really advanced technologies, especially if there is personalized data involved. There is also recommendations and actions involved. It's actually changing the database side of the information. For example, per regulations, you can't open a credit card without people really authorizing it themselves by clicking the link. If you really think about these in finance or many other industries, these consumer-based services are not perfect. You always receive these postcards asking you to open up a credit card. This is actually wasting a lot of paper. Now, like we've definitely seen, there is a lot of processes that you can actually automate through this information flow instead of this real paperwork going through. Why are these systems still not deploying? Some of them are deployed, for example, in e-commerce, which is relatively low risk. While with finance, healthcare, things are getting even more complicated. The bottleneck is on the compliance side.\n\n## Gaps in Current AI Agent Evaluation\n\nTo be able to pass compliance, the first thing people ask you is, how do you do evaluation? How can we do model checking? If you think about traditional machine learning, we have been doing this for so many years. We always create a benchmark. For example, like a question-answer benchmark would cause single-turn evaluation. You have a single query from the user, for example, how much is my balance now? Then you have an answer, which is standardized, which is a simple thing that you may be talking about a SQL query. You create this benchmark of question-answer, question-answer, and then make them into a static benchmark. Then you run your models on top of your benchmark. If you say I got 95%, I got 99%, so you have a number that you report to your model compliance team. You have a dataset that you can also give them to see this is input, this is models output, and this is the reference output, which is the ground truth.\n\nThis usually is how the model validation team works. It usually goes through the compliance later on. Nowadays, because the agents are multi-turn, you have a user asking a question that is related to his previous question. It's very hard to resolve this core reference problem. It's really difficult to really predict if the agent says something different, what the user would react afterwards. This makes single-turn evaluation obsolete. We cannot really evaluate with a static benchmark. What's even worse is there are tool calls that would change all the time. For example, the balance of your account may be different every minute. You cannot just have a static number that has a reference golden truth. Also, you want to look at if the agent is actually performing any of the actions, for example, if they're returning a product. You want to figure out if the product is actually returned or not.\n\nYou can't just look at the agent saying the product is returned. You have to look at the behind databases and the environments to look at whether the action was performed correctly or not, if the user's intention was met at all. This makes the evaluation of multi-turn agents or multi-step agents much more difficult than before. Currently, how are we actually evaluating these systems. What you would do is that you hook up a chat system. Once you build a chat agent, for example, the shopping agent, you would publish a simple web service, and then you're going to send it to your friends, your coworkers or your product manager to ask them, say, \"I built this agent. Why don't you help me to test it?\" What they do is they're going to chat with the agent and find all the errors that you have and then report it back to you.\n\nOnce you get the reported errors and you've got to figure out how to cluster them, how do I actually annotate, these are the same errors, and they have the same root causes, and based on that, I got to fix it. Then, I deploy it again. However, I can't use the previously collected evaluation data anymore because my agent changes, so the data is going to be different. What I do is I'm going to ask my friends again, can you actually help me to test it again? This process is really manual heavy, and it's a lot of time wasted of people trying to test your agent. What's worse is because these people are not your real users, they're testing ideas. They're pretending to be the users. Their user intentions are not realistic. If you have all technical friends, they're all testing things that technical people would think about.\n\nThat's why the testing coverage is very poor most of the time. That's why the biggest gap, you think, we did all these rounds and rounds of manual testing. We should be good to deploy. Once you deploy, you think, these users are really very different. They're actually talking about things I never imagined from a developer's perspective. This really comes into, can we actually have a better, more systematic way to manage these testings that is more multi-turn? Is there a better way to help people with different minds, closer to user minds, to be able to deploy this domain knowledge into the testing process to make things more iterative?\n\n## Solution: Synthetic Users\n\nOne thing that we develop in our lab is focusing on, can we actually use the magic to defeat the magic? One thing is, can we actually use agents to build user simulation so that we can actually run these agents automatically? We're going to say one agent is this user A, and there's another agent, it's user B, and another agent is user C. Then we can create a lot of interpolation of these different users so that we have a good coverage of potential real user distribution of your real user cases. Based on these different users for each of them, you can manage them and actually use these per user as one golden dataset for iteration that is more repeatable. This is how you can really think about, you have a simulated user, which is also an AI agent, and you have a product agent, for example, your shopping agent, and they would interact with each other to create trajectories that is similar to how your production trajectories will look like.\n\nBased on that, you can find errors and then start the evaluation process early on. Then, the good thing about this is that your user agent, in some sense, you can actually make it more constrained so that it will test different components or different functionality of your agents. In this way, you can spin up different kinds of user agents at the same time to do the testing in a scalable way. You can spin it up any time. Instead of you have to pin your friends in a group mode, how can you actually test my agent? It's relatively easy to deploy. Once you can think about, it's an endpoint. It's just an agent endpoint. You just talk to this simulated user endpoint through this API calling. You'll be able to generate trajectories of simulated data. You can automate a lot of the testing, and then you only needed a more important manual testing at the end of the day.\n\n## Solution: Arklex AI Simulator Demo\n\nHere is an example of how these tools could be useful. We have an open-source repo, it's also called ArkSim, that is specifically built for engineers to be able to do this automation on user simulation instead of you have to do it manually all the time. You can find all the errors before your product manager can. You can fix them and really ship a better version to the end. I'll just play this to help you to understand how the process will look like.\n\n\"We'll see an example shortly. You also define a list of agents' capabilities and the knowledge that agent has access to in order to generate the user goals. Together, it will be the scenarios that will be used in the next part to simulate conversations. Here is an example. I'm testing my Q&A agent for the GED test, which is equivalent to earning a U.S. High school diploma. I can customize my simulated user's education status to less than high school diploma. Prepare your agent's capabilities by describing what your agents can do. Also prepare the knowledge file by adding the corporate overview, business type, and a set of knowledge your agent has access to, such as frequently asked questions or website contents. Here are additional configurations such as the number of conversations, number of conversations per persona, when generating the scenario. After we prepare the scenarios for our agents, we can run the latter two parts in our CI pipeline to start to simulate and evaluate the agent.\n\nThis ensures that we test on a fixed set of scenarios to control the testing. Here is the codebase for my agent. Now I've made some changes to my agent and I'm ready to push them. When I push the changes, my CI pipeline will start building my agent with the new code, and it triggers the build for running the simulator with the new agent. It will first spin up your new agent inside this build. Then it starts to simulate conversations with this agent. We can override the config directories, number of conversations, and number of turns in the arguments to customize the simulation. Then it continues to evaluate those simulated conversations and identify issues in these conversations. You can use a threshold value on the conversation score and this will signal if your build has passed or failed. We also provide a more detailed evaluation report. It shows scores on five dimensions including helpfulness, coherence, verbosity, relevance, and faithfulness.\n\nWe also summarize the errors in the agent behaviors. We categorize them into five categories, fail to ask for clarification, repetition, disobey user request, false information, and lack of specific information. Here is an example. In this error found, it's categorized to failure to ask for clarification. We show the explanation of this error and the suggested fix. Clicking into the conversation, we can see the more detailed user goal, the scores of the agent behavior in this conversation, the simulated user profile, and the goal completion score explanation. For example, in this conversation, the agent successfully solved the user's query so it gets a goal completion score of 1. Diving deeper into the conversation, we highlighted the agent behavior errors on each turn. Scrolling down to the bottom of the report, we can see all the simulated conversations and evaluate the performances. By integrating the Arklex agent readiness tools with our agent building CI pipeline, it helps developers reduce manual agent testing time significantly and signals your agent readiness for deployment and production early.\"\n\nIf you're interested in using the open source, you just search Arklex AI ArkSimulation, you'll find the open-source repo.\n\n## High-Level Architecture\n\nOne thing that we really see is like, as we say, how do you define what's the input of your user simulation? Depending on the agent process that you design, for example, you can actually define different attributes of the user. If you're doing shopping agents, then the attribute of the user is really about their demographics. Are they budget conscious, or are they budget non-conscious? These entry points that are usually controlled by the product manager of all the operations people. Then these are usually what we say, like you use this kind of information to create your user profile. Then the next element is what we call user goals. User goals are basically what the user wants to do, or sometimes we call it user intent. The user intent is actually seeded by the agent's capability. Based on your agent's different tools that are enabled, different sub-goals that they can complete, you can actually combine them together to form different kinds of goals.\n\nThen the third component, we call it context information, because relevant information sometimes is like documents related to your product, or information related to the goals that the user wants to complete. These are the three components that you can actually change in different values so that they will permutate to get a lot of different coverage of the scenarios. These three components combined together is called each individual scenario. You can treat that each individual scenario is corresponding to one user simulation. Now you basically have all these scenarios you can manage over time, and that you can connect to your CI/CD pipeline so that whenever you are changing your agent, you always want to run these simulations based on these scenarios that you identified are critical to you, that is simulating how your real users would behave. Then once you run these simulations, as we said before on the demos, you'll generate these conversations.\n\nYou're basically calling the different tools, and based on that the user is basically fulfilling its goal, given its persona, taking the contact information it has, for example, like the conversation before. Basically, your agent's simulated user and your product agent are interacting together as this is a real user testing your agent. We provide different API tool callings for the user. The simulated user can also do GUI actions as well, so click on different parts of your webpage and so on to really simulate how the end users would perform. For example, as we said before, the Walmart Sparky applications would provide different product cards. We can actually simulate how the end user would click on different product user cards so that you know like, whether these product cards are actually having errors on opening and so on. In this way, they can have a very close to real user experience testing using these AI agents.\n\nOnce you generate all these conversations, the most important part is, how do I find errors within these conversations? This is what we call evaluation. In general evaluations, many people would use what we usually call large language model as judge. Language model as judge is very useful because it's utilizing the large language model's internal capability. The most important part is actually evaluation has to ground it towards the task. You can't just imagine the large language model knows everything about your applications, your documents about your customer service in order to find your errors. Here, the nice thing about simulation-based evaluation is you know exactly what the scenario is. You can actually do the verification, which is evaluation, in a much more fine-grained way. For example, my goal of the simulation was, I'm a 20-year-old, I just bought a dress. I wanted to return it. Then the corresponding evaluation of the task completion can be automatically verified as, did I actually get the product returned?\n\nDid the returning actually follow the principle of the particular business? This can be automatically enumerated through automatic rules to check whether the end goal is completed or not according to the correct process. However, you're not doing this only by one goal, which is task completion. You can also design different metrics that is turn-by-turn just to see if the agent is actually performing things that you really wanted to expect it. For some people, some of the turn-by-turn metric is useful, for example, helpfulness. I want my agent to be more helpful than not making errors. You can use this helpfulness to specifically design an LLM-as-judge with more context, more examples. Then, you can also have one, for example, adhere to brand voice. Your brand voice can be very different. You usually use natural language to describe what your brand voice is. The hard part is you have to have some hard negatives and good positives to give the large language model some seeding to understand.\n\nUsually, if you don't have any data when you're building that agent, it's very difficult to find these examples. Using simulation is great as I can simulate these conversations and involve your experts to really look at the generated conversation to say, this is actually a hard negative, please involve that into the LLM-as-judge in the evaluation. Simulation is giving you a lot of what we usually call synthetic data to help you to build your evaluation in a more grounded way. Also give you more visibilities about what the real user would interact in the end of the day, what the real conversations before your deployment should look like.\n\n## Identify Agent Issues\n\nHere is an example of a shopping agent. For example, this user simulator, the synthetic user is saying, I want to check my order status. Here, when we simulate a user, we usually give them a user profile. For example, in e-commerce, we definitely would give it various different kinds of user profiles to test the tool applications of the agent. For example, now, this particular user has two orders in its user profile. That's why agent would say, I see you have two recent orders. Order 1299 has been fulfilled, and it is included. Order 1300 is still pending fulfillment. Which order would you like to check? You want it to do the clarifications. This is one type of more complex situation. You definitely want to make sure that you're doing the testing. Then the synthetic user, according to its goal and its profile, is going to say, when will order 1300 ship?\n\nThe agent will retrieve and do all the things that you do, and say order 1300 is still pending fulfillment, which means it hasn't been shipped. Then at this point, as a synthetic user, according to a lot of the data we have seen, and according to some of the scenarios we'll give it, what would the synthetic user do? Here we say, can you tell me when it might arrive in Seattle? He is still trying to do it one more time, and giving more detailed information, giving its profile, in order to be able to get information from the agent. The agent now we're testing, coming back saying, I'm sorry, but I don't have the exact shipping order for blah-blah. I can't provide an estimate. Then, at this point, for the synthetic user, given the context, given the profile, he's going to try to say, can you give me the customer service email?\n\nThis is how a realistic user would usually say. Of course, we would actually also simulate different users with different emotion type as well. The agent would say, I'm sorry, but I can't provide blah-blah. With this, like you would see, this is a complex situation. The agent will capture some of the agent errors. Then based on the goals, we know that the user didn't get what they want. Then we can base on their reactions, and really post on what potential issues that agent's already experienced. For example, here, we see that agent cannot access the shipping information. Then the natural thing we say, it might be that some tool was broken down in terms of accessing the shipping information. Then there was like, it couldn't find customer service email directly. There is some lacking of information in your RAG pipeline or in your vector database, or because there was some confusion between different information on customer service that resolve this confusion for the agent in the end. For testing, when we say evaluation, it's not always just for the evaluation sake. It's really to find problems and being able to shed light on how to improve the agents in the next round.\n\n## Evaluate Simulated Interaction Quality\n\nHere, before I move on, how do we improve? I want to also talk about how do we evaluate the user simulation in itself. We just showed you an example of how the user simulation works. Of course, we wanted to know how can we also quantify these user simulations, basically attach some numbers towards it. Then what are some metrics a good user simulation should be able to follow and optimize towards it. If you think about it, when you're building an AI agent, everything you would do during optimization is increasing your task completion. Basically, make the agent more productive, and being able to complete tasks more correctly. The optimization for the user simulation, which is the user agent is actually different. We wanted to be able to have more coverage, which means that, for example, it will be able to invoke different tool trajectories out of the agent that has a lot of diversity.\n\nThis is what we call a tool call distribution entropy. Then we also wanted to see how your agent would jump from one tool call to the other tool call. This is what we call tool call transition entropy, we also wanted to maximize. Then, of course, the end was what we call trajectory distance. We want to make sure that all your potential trajectories are actually not close to each other. For example, when I want to return a product, I'm also always just part of one thing, and I want it to immediately return. This is not giving you much of a coverage. We wanted to simulate different users, who would buy different products at a different time. Some of them are returned. Some of them are not returned. All this complexity and coverage has to be there so that you can have all these simulation scenarios that are more complete.\n\nThe second part, of course, the most important part is what we call realism. If you build a user agent that is just doing adversarial things to trigger agent error, that's not enough. You wanted to make sure that they are doing these things and they are also your real users at the same time. They're not only attackers trying to trick your agents to do bad things. You also wanted that these are real users having real problems and behaving the way that the real user will look like. You can use various optimization techniques, for example. You can have reward functions adhering to the profile, adhering to the goals, and also distribution similarity from realistic data that you collected that is close by in your scenario. Then the third part is, of course, you don't want to make it so expensive to run this kind of simulation-based testing all the time.\n\nFor example, you always simulate 10,000 conversations every time, and then every time you go through your CI/CD pipeline, you're costing a couple thousand dollars, which is not really ideal. One of the things we wanted to do is really use the minimum number of scenarios to give you the maximum coverage. In that way that you can run them in a more efficient way and reduce the cost that you really have to shoulder. Then, finally, the most important downstream evaluation metric we're really looking into is agent failure identification. Basically, how many errors can we find in a given product agent? For example, one product agent, let's say we treat it as a black box. It's the Walmart Sparky agent. We would have different versions of the user agent. You can try different ones. We want to really be able to find this kind of simulation that has big enough coverage that can trigger different failures so that we can find these errors before the real users can. Eventually, this is the most important metric after all these three more NVIDIA metrics we evaluate.\n\n## Evaluate Agent Performance (LLM-as-judges Continued on Scenarios)\n\nFinally, once we say that we have a really good user simulator that is hitting high on all these kinds of metrics, then we're also looking into what is the evaluation metric for the product agent. As we talked about before, sometimes if it's a customer service, then you really want to define things towards your product goal. Customer needs completion is usually what we usually call, it's usually 1 to 0, you can normalize it. Then, for example, efficiency is also important. You don't want to waste your customer's time. Then you also want to see, for example, for some of the interactions, it's a go-to-market agent. For example, I want you to subscribe to our mailing list. Then the call to action of the agent, which is basically the agent wants to persuade the user to do, is also important to measure. Depending on your use cases, your agent's final evaluation performance metrics should be also different. You can actually automatically deduce that from the simulation and use that to reduce this cost of extra design.\n\n## An Ever-Evolving Golden Scenario Set\n\nFinally, if you deploy your agent after going through all these simulations, you're going to generate a lot of production logs. Simulation is never perfect. As we can see, for example, in self-driving cars, you're always finding new corner cases every day, because there are new roadblocks there and here. In real deployment agents, the same thing. Your user distribution may change and it's always ever-changing. For example, we talk about e-commerce selling agents again, there is new holiday comm. There is a new policy about the promotion comm. Your user might behave differently than before. It's also important to update your user simulation over time so that it's closer and closer to a real user distribution. What we do is also you take your production logs and try to summarize, given your existing simulated conversation, what are the differences between the two distributions? So that we can create additional simulation scenarios to cover the missing piece.\n\nBasically, you can induce the extra one that we haven't covered in terms of attributes of the user, in terms of user goals and then user context, so that you can actually evolve your CI/CD pipeline's golden scenarios. In this way, you keep evolving and making sure your agents are always updated. You can imagine, for example, if you ever worked in consumer-facing agents, the only thing that does not change is the users change all the time. Maintaining the agents is actually a huge effort for all the operations team. Having a mechanism that you're actually optimizing for your user simulation and really reflect into your entire engineering pipeline is the most effective way to help you to reduce the maintenance of your agents over time when you're deploying.\n\n## Enterprise-Focused with Proven ROI\n\nWe worked with various different companies. For example, one of them is Pearson. Pearson is one of the largest education companies. We really care about making sure the students are getting their customer service or education consulting in time. The problem is students speak different languages. They serve more than 200 countries. Every individual country has different testing policies and purchasing policies. You can't really release these without doing testing on each individual sub-module. This is one of the biggest pain points for them. They used our simulations to make sure that things are much more rigorous before they're actually deployed. Though we have been working with them for the past two years, the agent really is optimizing over time based on more initial data analysis and an automated loop of simulation optimization. Another thing that we really care about is definitely the continual learning. How can we actually make sure this is a flywheel that is automated, instead of there's a lot of human handling and we have to do data analysis on our production log and then manually find these bugs and prioritize which one is the most severe bug, and then try to come up with an engineering pipeline to fix at what time.\n\nA lot of times, if you think about these agents, there are a couple of components we always see. For example, it's like the flow of the agents. Which tools are available at which point? Then there are the prompts. These are basically just two optimization points that we can optimize using AI as well. Let me just show you one of the examples. Let's look at the before. This is a customer service agent. As we said before, if you are not very experienced in building these agents, what are you going to give it is probably this very simple flow. If a tool call says like get user details, the intent is like return order. Here basically the user says, I want to return my order. Then we detected that the intent is return order. Then if you say the intent is return order, I'm going to just do the tool call, return order.\n\nThen we find actually there is no order that can be returned. Sorry, there is nothing to return. Then the user gets into the confusion stage and this conversation fails. The real reason is because the product hasn't shipped yet. You can only cancel order, you cannot return order if this thing is not delivered. The user doesn't really know. He's just saying, I didn't get my order. I want to return it, because this is the way that they would express it. This particular corner case will always leak out. The realistic things that we would find and the right one to optimize it is really on the right one. Is that, if you find people saying, I want to return order, internally, you should go into whether if the return order is not triggered as an error, it should fall back to cancel order to figure out if there are certain things already purchased but hasn't been shipped.\n\nThen, finally, you would have come into the natural conclusion. Your flow shouldn't be rigid as the left one. It should be able to expand it to the right one. Being able to test these are the corner cases is the first point for you to be able to optimize in the correct one on the right. Right now, if you think about the user simulations trying to find these corner cases to really trigger the error, and based on trigger the error, give you suggestions on how to improve your prompts, your user flows as well. Of course, you can also in some sense improve it by training. Based on the synthetic data you generated through the user simulation, you can fine-tune smaller models to be able to actually really look into the successful one and also penalizing the unsuccessful ones to make it more efficient, smaller models to serve your end user if you have a huge volume amount for these customer facing agents.\n\n## Do We Also Need Simulation for Workflow Agents?\n\nSo far, we just mostly talked about these very interesting customer service agents, having back and forth interactions, but there are also workflow agents that don't really require input and output in natural language. Sometimes there are workflow agents. For example, I'll give you another very interesting condition, which is mortgage underwriting. If you have applied for a home mortgage before, you have to supply all these documents. At the same time, you also need to fill all these forms in order to give the underwriter to be able to go through this process. The underwriter, its main goal is to find inconsistencies of these documents, or sometimes they're PDF. Sometimes there are images because you lost the original one. Sometimes there are filled forms. Some of them are unstructured text that they pulled out of your credit histories or whatever. This is really a problem in terms of we have all kinds of different modalities of input.\n\nThen we're trying to do some formal workflow in order to get out the outcome, which is, if this person has risks, do I need them to actually supply me more information in order to resolve these risks? This has usually been done through manual human labor work in order to find these inconsistencies. This is actually a great opportunity for AIs that can actually ingest all these long-form documents through various different modalities. One of the biggest problems is we don't have data to train these agents because these data are all private information. They're PII restricted. You can't really share it even internally to your production teams. That creates a huge problem and bottlenecks. This is also why simulations are not just for testing. They can also be for training purposes. For example, we can synthesize starting from this one user, we can manually give this user all the enriched attributes that are consistent with itself.\n\nFor example, this person's SIN number, their employer and so on. Based on this information, we can manually, using AI, fill these forms that are required for home mortgage, for example, and generate synthetic paystubs, bank statements or everything that is consistent with this user. Really importantly, create corner cases of these underwriting rules. You can go through the underwriting handbooks and find all these corner cases in the underwriting's head. We use these examples to create these synthetic data to have a good coverage of all potential corner cases. In this way, for example, you can have these T4 forms. You can also create all these obstacles that are similar to realistic behaviors. For example, you can stitch documents together in the wrong way, like missing certain pages of documents. You can get a tilted, skewed, blurred, poor lighting. This is what AI is good at. We can use these pipelines to generate these variations of coverage of synthetic user. It's not only about what they say, but about this meta information, their documents and so on. You can actually create this perfect digital twin of your user. These are not real users. These are synthetic users trying to have all the corner cases of different borrowers. In this way, you cannot basically automate the process.\n\n## Self-Learning Agents\n\nToday I'm mostly talking about this as, everything right now, if you think about the user simulation or the tool simulations I'm talking about, in order for you to do the testing, they're not just for testing. They're just for improvements as well. Because the first thing is that you've got to figure out how your real users do. Before you deploy, you don't know how your real user does. You can use a simulation as a surrogate to help you to do the improvement before you can do anything better than that. Then you deploy. Then based on the real user's action on your real production environment, you can actually further iterate to improve your different tools, your different mechanisms, for example, better structure of memory, better structure of your tools and better structure of your prompts. In this way, you can actually do various different things automatically or semi-automatically to improve your agent's quality.\n\n## Questions and Answers\n\n**Participant 1:** I'm curious about how far the age simulation will go. Can you simulate, say, a toddler calling that shouldn't be and getting a reject, or a senile person calling and repeatedly forgetting what was just said, or something like that?\n\n**Zhou Yu:** You could simulate all these now through existing data and also trying to give it some prior knowledge, domain knowledge of the real operators. Through this, you can simulate different users of doing these profiles, doing these intents, and doing these corner cases. Then the most important part is you really let the simulation play out the scenario so that you can see what the real trajectories look like. Then you involve your real experts to look at them to see, are they actually experiencing the same thing. For example, like customer service agent. Then they can read through this and then give natural language feedback, and then iterate on in order to make the simulation more perfect.\n\n**Participant 2:** We were talking about that the evaluation should happen around something that is easy to verify. For example, was the order canceled or fulfilled and so on? What do you do about less defined targets, for instance, if the user asks to describe a product or to compare two products?\n\n**Zhou Yu:** Some of the scenarios that the user has a very specific what is considered the completion of the goal, for example, getting a product returned. Some of them, it's more soft, for example, like I wanted to get a recommendation of a certain product. The recommendation usually also depends on the profile of the user. At the same time, because we make the user to be able to also get access to their own histories, we basically have a model in terms of basing on the previous history, given the recommended product this product agent recommends, what is the reaction of the user? Then based on that, we give the like, I will accept your recommendation or not. If I accept your recommendation, means that you're doing a good job. If I reject your recommendation, means that the agent's optimization could be better.\n\n**Participant 3:** When training models with different capabilities, so simulating someone who has or does not have a certain background or context, do you train new models from these? Do you just put an overlay on existing models? How do you make sure that these are accurate representations rather than caricatures or just particularly salient examples of some of the weirdest most distinctive behavior?\n\n**Zhou Yu:** We usually separate the simulation into two types of scenarios. One we call cold start, there is no data at all. The other one is like we have warm start. We have some production data from the humans who actually handled these tasks before. If you have warm data, it's obvious. You wanted to just do the simulation that matches the distribution on every aspect of the real user. The hardest one is always the cold start. The cold start is the hardest, the only thing that we can do is to really elicit the domain knowledge of the experts who have seen some of the real experience before. In this way, we ask them to conduct interviews in order to try to figure out where the space will look like. Based on these interview data, we'll create these initial simulation scenarios. Then, once they run the simulation, the expert would give feedback and do iteration the best they can. Then after that, they deploy, then we get real data and then we refine it.\n\n**See more [presentations with transcripts](https://www.infoq.com/transcripts/presentations/)**", "url": "https://wpnews.pro/news/presentation-from-ai-agent-demo-to-production-automated-testing-and-evaluation", "canonical_source": "https://www.infoq.com/presentations/ai-agent-testing-evaluation/?utm_campaign=infoq_content&utm_source=infoq&utm_medium=feed&utm_term=global", "published_at": "2026-09-07 11:00:00+00:00", "updated_at": "2026-09-07 11:25:23.530209+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-agents", "ai-products", "ai-research"], "entities": ["Arklex AI", "Columbia University", "Walmart", "Sparky", "Amazon", "Rufus", "Zhou Yu"], "alternates": {"html": "https://wpnews.pro/news/presentation-from-ai-agent-demo-to-production-automated-testing-and-evaluation", "markdown": "https://wpnews.pro/news/presentation-from-ai-agent-demo-to-production-automated-testing-and-evaluation.md", "text": "https://wpnews.pro/news/presentation-from-ai-agent-demo-to-production-automated-testing-and-evaluation.txt", "jsonld": "https://wpnews.pro/news/presentation-from-ai-agent-demo-to-production-automated-testing-and-evaluation.jsonld"}}