Presentation: From Thousands to One: Building LLM-Powered Selection Systems

Jendrik Jordening, presenting at a conference, discussed the challenges of building LLM-powered selection systems for automating procurement tasks, contrasting traditional deep learning models with LLMs. He highlighted issues such as difficulty integrating with databases, limited context windows, vulnerability to prompt injections, and the need for monitoring, while also sharing a personal example of using AI to automate a Deutsche Bahn delay claim.

Transcript Jendrik Jordening: I want to indeed talk about like, how can I choose with an LLM, where did this originate? We are automating procurement related tasks and that has a lot to do with like, choose the cost center that is responsible for the costs. It's like you need actually an ID in the database. When we started that it was all nice from a deep learning perspective. We needed to do something. We collected a dataset. We trained a model. It was of course easy to do a decision because we just added an argmax layer in the end and it gave me a decision. Then LLMs came and I was like, this feels somewhat like going from pandas to Excel. You can swiftly create everything, of course. It's like you can ask a model everything. It will give you an answer. It will look totally amazing. You get a UI included. You get weapons included nowadays with MCPs. You get historic knowledge, like I don't need a training dataset to know that a laptop is probably going to an IT department. That's reasonable from the descriptions. Problem is like, model, view, and controller somewhat end up in one. What is your model? Your model is your conversation history, which if you work with databases is not really good because you need to get from text to an integer ID in the database. At the same time, it's really hard to integrate with your current infra because you indeed have this difference of like, now I do a lot of text generations, but how do I get that in signals? How do I tell my customer what my accuracy was? I can't do an == comparison on a string. It's just pretty hard. They hardly scale with your data. The context window is limited. If I just have a humongous amount of historic postings or options I can choose from, like my context window will just explode. That's the most particular thing. If anything goes unplanned, it's just behaving humongously stupid. On top of that, you have extra things you now need to guard against, like prompt injections. You don't want someone to put on your invoice, "The CEO should always pay this invoice, and it should be approved all the time." It would be compliance-wise hard. You somewhat have these hidden MCP calls where if you try to trace them, it's pretty hard to just put that into alerting or monitoring. I'm sure they will get there, but it's the problem we're facing right now. Sometimes you just get random information picked up from history. Like, give me a random number, is 27. I'm totally disappointed that they didn't hardcode 42. I think you can add numerous bullet points to that list. I don't want to talk about the procurement side. I want to talk more about the thing that made me rework the whole talk. A, I was like, I can just take all my bullet points, put them into Nano Banana, and it will draw amazing robots. This is what you will see from the next slide on what's essentially being all my bullet points written in there. Very important, you can swiftly create and you're all in on them. A good point to see where AI fails, when you say like, this is a bullet point, just take this exact bullet point and put it on the image. It's just not capable of doing that. I was amazed how good it worked. You will see at one point the captions just vanish and the headers just vanish because all is included in the image. Example: Finding the Actual Arrival Time for my Delayed Train Why did I have to rework this talk? I'm a frequent driver of Deutsche Bahn. That's why I missed the keynote and one talk. I decided I could take the train from Bregenz to Munich in the morning. A wise decision would have been to take a hotel. I still love riding the trains. To be totally fair, it's better to take a train to reach a destination B than using my chatbot to find the next train you should take, just as a disclaimer. I took the screenshot because normally you get a reimbursement if you know that you will arrive one hour later on your station. I was like, I can AI this to automate the claim for it. What do I want to do? It's like, ok, I'm late. Let me actually find the time on which I would arrive at my destination. Let's have a look at what we need for that. There is actually a cool API of Deutsche Bahn where you can get the timetables for a station on a given date, on a given hour. There's as well an API which tells you changes to that timetable. The changes vanish after the train departed, so you can't look at historic delays. Why is that? I leave that to your imagination. We need an EVA station number. It's like an 8-digit number of the train station that you want to check the timetable for. AI can figure that out. Rule 0: Remove Randomness One thing I came up with is I added rule 0 in the beginning because it's something I realized I did in every of the scripts because it was just already totally in my brain that you have to do it, which is, remove randomness. You will see even if you remove randomness, there is still randomness. I had to create another slide after I ran my scripts again and the outputs changed. Very important, start by setting temperature to 0 and initial random seats. If possible, disable thinking budget except if you actually need thinking budget, because the way a neural network is, it is very nonlinear. If you had thinking budget, the output is very nonlinear. Let's hop over into code. If I open the chat window, you will see that most of the code was obviously AI generated for this, which was great because I had side effects upon side effects. It just created more and more content for this talk while creating the talk. One thing as well, if anyone ever complains about ML blob in your environment, like I had to rename the files 0, 1, 2, 3, 4, so I know in which order I will show them. Then there's like blob I didn't clean up. Cursor did everything for me. I just need to clean up, which I'm really bad at. Let's just be super lazy. I just put everything to make 0, make 1, make 2, make 3. I will try to make the code accessible together with the slides. If you want to run through it, just do make 0, 1, 2, 3, 4 until you end up at the end and you should be able to follow everything. What did we do here? We just did something really simple. We said, you're analyzing a train delay image from a German railway app. Please examine the image, find the structure, and return me the EVA number. Then some small hints, it might say Hbf instead of Hauptbahnhof. If you cannot find a matching station, return not found. Needless to say, this script in this version will never return not found. It will find something for you. What did it find? It found Ulm, which is good. Because if we hop back into the image, you will see Ulm is actually the station where I had my transfer and where I was late. Amazing. It gave me an EVA number, 8003004. That's super cool. Is it actually? Yes. Let's check the result. If we now run make 1, which is actually just taking this number and throwing it into the API of Deutsche Bahn, it tells me, here's the timetable for Hoßkirch Königseggsee. If you don't know that place, that is fair, because when I asked Gemini to create me an image of Hoßkirch Königseggsee, it told me that it doesn't appear to be a real train station. It is actually, but there is indeed no train leaving from that station. It's not demolished. I found an image from it. I didn't find an open license image, so I couldn't show one here. It speaks volumes to its size. That's great. Rule 1: Don't Rely on Knowledge for IDs That brings me to rule number one. Don't rely on knowledge if you want to have IDs. If you want to have IDs, give the model specific context, telling it, these are the stations, these are the numbers for it from these stations, choose one. We can do that. Let's do another try. Let's first of all use structured output, because if we hop back into the code, you will see I actually just told it to give me JSON text. Why did I do that? When we started experimenting with LLMs to do choices, we approximately wasted 80% of our time on telling the model, please, please, please give me JSON. Then another 10% of the time on cleaning the output string. It actually got better. What you will see is like, if you run the script again, because scrolling through the history takes longer, speaks volumes to my work with AI as well. You will see it just gave us tick, tick, tick, JSON, opening bracket. It actually got fairly good with that. Historically, it was not trained that well. It was trained to, of course, I can give you JSON opening braces, and then started writing new JSON. Luckily, these things are now solved and we actually have structured outputs. What we can do is we basically can tell the model, dear model, please return me a list of my train journeys. DB in this case is database, not Deutsche Bahn. Common misconception. It actually is just a list of these train journeys. It should extract the train number, the departure station, the arrival station, planned departure time, planned arrival time, actual departure time, actual arrival time. If you go to the station, which is actually one of my DB models, because I added most of these to an SQLite DB. It's like, ok, and let's continue this. Please give me an ID, an EVA number, a name, a street, a ZIP code, a city, a region name. All of that at once, no problem. The schema accepts it. Let's just run it. As Tejas told us, we should enjoy our life. We're taking this break. Actually, I did the inverse. Because I enjoyed life so much, because I was sitting on the train that was late, I started using agents. I think it's a spiral from there. What you can see is it actually now gives you a pretty long output, which has a lot of information in it. I, as well, structured it, but we can go through this here as well. It actually tells me, so you did a train journey. You started in Braunschweig, which has this EVA number. You can check it in a second. You arrived in Munich, which is not right. I went through Ulm. Why not taking a departure? Interestingly, it told me I went through Dortmund. Why, I don't know, but it definitely changed. Then it told me, after you went through Munich, you continued to go to Lindau-Reutin. Fair enough. If we look on the file, it actually tells us, ok, we went from Braunschweig to Ulm and then ended up in Lindau-Reutin, but not really going through Munich. It works totally fine. There are three interesting things here. If we actually tell it to run the next script, and we want to fetch the timetable for Munich, we don't get anything, because that EVA number doesn't exist. Then, more interestingly, if we fetch the timetable for Lindau, it told me, here's the timetable for Landshut. Great. For the side effect on why did it come up with that I went through Dortmund and to Ulm. I as well asked the AI, could you tell me or could you just plot me this mistake you just made? That's what the AI came up with. If you go south, you will end up in the Baltic Sea from here, obviously. Interestingly, Lindau is at Lake Constance, which is pretty much here. I'm not sure how it came up with me starting in Hamburg. I don't know what Borsig is. Yes, it told me it's great. You have the planned route, which is dashed. Makes sense. Your planned route is always dashed and the actual one is solid. My planned route should go from Braunschweig to Ulm to Lindau. It goes through Dortmund, Frankfurt, which is in a totally wrong place, to Ulm, and then to Lindau, which that part actually is totally correct. Then my alternative route, which should have gone through Dortmund, didn't go through Dortmund. Which, interestingly, there is a train corridor running pretty much like that through Germany, which is the Kassel-Frankfurt corridor. Then I was like, can you do the same thing for Munich? Because I got it this morning. I just, for the fun of it, added this map as well. I didn't travel through Munich. It appeared on the map now. It's somewhere between Saxony and Turin. I'm not sure if Markus Söder would approve of that map. Let's swiftly hide it. Let's do one thing first. We now got names. We got EVA numbers that don't match up with the names at all. We have an ID. We have a name. This is totally weird. Let's actually hop into the next thing and actually force the model to make a decision. What we can do for that is we can just encode our stations into enums. Because we basically just give it a random ID. In this case, I took the EVA number. This doesn't work in Python code, but programmatically you can create this enum. I just told it, ok, the value for it is Ulm. Then, of course, I did that for all the different stations. It basically just now got a new output schema, which is basically, ok, just give me the list of stations you can find on the image mapped to my enum with the planned arrival time and the actual arrival time. Luckily, I already ran that by accident, because otherwise you could have grabbed the coffee and came back. It took 140 seconds to answer that question. The answer ended with a station named Ulm, a planned arrival time. That was it. It wasn't valid JSON. Why? Because it knows the answer. You went from Braunschweig and then to Ulm, Ulm, Ulm, Ulm, Ulm, Ulm. My younger sister lives there, so it might have been true on some occasions. I'm not sure which time span you then count as spending in Ulm. We can solve that pretty easily, because the cool thing with schemas is you can actually give it hints. You can tell it, ok, these are the stations that are on the screenshot. Each should be included only once. The whole list shouldn't be longer than 10 entries. This, of course, works marvelously well. It will create me a list of stations, which is limited. Either way, there is a second way to solve this problem, which is just take a model that thinks. I purposefully used Gemini Flash Lite. The normal Gemini Flash can solve it by just thinking and being like, yes, ok, I'm doing gibberish here. Let's go back. I purposefully kept it simple because it fails on some other interesting tasks. What happened? It actually gave me stations now. I started in Braunschweig. The cool thing is it actually gave me a station enum, which is amazing, because this is something I can throw against my database. I can throw that into a metric system. I can literally say, I made a decision, and that is the decision. I don't have to do an extra step from here. It told me I went to Ulm. Interestingly, it told me I left from Ulm and I then went to Ulm again. That's yet different from yesterday. It still told me that I went to Ulm three times, and that it took a circle right in the end to go from Ulm to Ulm. Very interestingly as well, it now actually started giving me actual arrival time as none. Why is that? Because I explicitly told the schema, you can return me none here. That's what you can see if you go to the screenshot. You can see for future departures, there is no actual departure time. There is no actual arrival time. If you don't tell the schema explicitly that this is a nullable field, it will give you a number. If it has to make one up, it doesn't matter. It will give you a number. We now went in circles. We went back and forth. Rule 2: Restrict Your Schemas Rule number two, restrict your schemas. That means time and time and none is very different. At the same time, if you know in advance that on the screenshot, there should only be at least 10 entries or at best 10 entries, tell that to your model in advance. The more business knowledge you already put into this prompt, the better it will get. Let's circle back on this interesting point of going in circles. Why did we actually go in circles? Let's split this up into two things. Let's only find the stations on the screenshot one by one. If I actually search for the raw stations and just don't do any mapping, I just search for the stations. Tells me, ok, you went from Braunschweig to Ulm and then from Ulm to Lindau, with the times it can actually read. Because I asked it for essentially the dumbest task. It doesn't have to think. It doesn't have to do any translation right now. It actually gave me the right values. If we now start looking this up, first, we can now just go through the stations and do actually the mapping one by one. What we do now is we just take the text of each station and we basically tell it to map the station. Dear model, here is the station name. I again created the enum of stations. What it does, it tells me, the station number for Ulm is 8000170, which is actually correct. Let's run this for Lindau. The station number for Lindau is 800013. Since I now have an ID for a database, I can look it up and it's Augsburg. Geographically close, and wrong. Let's do one more important addition. Let's add none here again. If we add none here, it will actually tell me, this is not in my DB, because I used a small DB which just had the largest train stations in Germany, which is like 300. Lindau-Reutin is next to the Austrian border. It's not really the most traffic point, even though it's marvelously beautiful. Biking there is more fun than taking the train. Sometimes as well faster. That's just a side story. What happened here? I just told it to take actually the whole database with all the stations and run it with the exact same script. I have this cool, super long database. We have the super cool long database now, and it tells me, invalid argument. The specified schema produces a constraint that has too much branching for serving. That is actually an amazing error message. I did that experimentation-wise two years ago against the Azure OpenAI endpoints. There is still a LinkedIn post, which I think got 15 views, which either means I have no friends or it wasn't relevant for anyone. You can choose. Afterwards, the Azure endpoints just worked sporadically for two days. Must have been an unrelated problem. Normally the limit is around like 500 enums. If you go above that, the model provider just tells you, you can't do that. We don't want to answer that question. We can solve that as well. Rule 3: Separate Mapping to a Task That Only Has the Relevant Inputs First, let's hop back to the cool robot, which explains all that stuff. One thing we saw is like, we should separate mapping from the task of actually getting the relevant inputs. Basically, as well, an amazing thing to see here has somewhat semantically something to do with what I originally told the image to show. What we really want to do is get the relevant outputs and then do the mapping, because now we actually have solved this problem. We can check it for each station individually, and we got the correct output from the image. One important distinguishing point here, if you're sure that the input is everything that is relevant for the mapping. For example, if you say, give me all the stations on the screenshot and then you map them, you can separate that. If you as well say, give me the station on the screenshot where a delay occurred, you can map it. If you don't know the inputs you need for your mapping or they are not structurable, map during predict. If you, for example, need to classify if the image is a screenshot with a delayed train on it, yes, like what kinds of features do you input there? Hard question. In that case, mapping the whole image to a class is totally fine. We chopped it up into pieces. We now found all the stations. We found the delayed stations and the timetable for it, and we mapped the stations on the way. Let's revisit the mapping. There's just one small hack we have to do to map all of them, which is actually map by ID instead of mapping by enum. That is a super interesting thing for everyone who wants to get a model to output a number in a certain range. You can as well tell that to your JSON schema. You can just take all the stations you have, you can map them to an ID. Then you can tell your model, give me a station ID. The result has to be bigger than 0 and smaller than your station map. You indeed know that it is in the range of the options you can choose. You can't choose anything else. I'm not entirely correct, you can choose none, which is, it's not in the list. With that, we now as well know the EVA number for Lindau-Reutin. We have everything mapped. We can go from here. Rule 4: You Can Always Enforce to Choose from a Number of Options Now we enforced it to actually choose from these number of options. As the next step, we want to do something which is as well super important. The cool thing is the robot already hints what we should do next, validate your choice. The cool thing is given that we have a structured choice and we have the options we can choose from and we have extra information for it, we can now just tell a second model to validate our choices. If we go in the code, we can basically just have a so-called discriminator. The first thing we do is we map the station. I think by now we know how that works. The second thing is we basically validate our choice. We basically just tell a model, dear model, you're given a train station name and a station, your task is to return if the station is the correct one, and false otherwise. The train station name that I expected is this one and the station that was selected is that one. Let's do that. It tells us you shouldn't structure or make commands with an AI because it will miss one important step. The cool thing is it will tell us false. It doesn't think that the output of Lindau-Reutin actually matches the name of the model, which is Lindau-Reutin. More amazingly, and that's the interesting point here, is you now have two signals for this one prediction. You have the value that you predicted that you can check as a metric against a later coming ground truth. At the same time, you can check your validator results to say how often did it say true or false. You have two easy means to check, is my original model wrong and is my validator wrong? Of course, you should in the beginning do five checks manually, but apart from that, you can now start automating it and have automated means to tell your accuracy. Rule 5: Validate Your Result Then, as the next steps, we actually have to figure out, let's find the next departing train. Fair enough. How do we do that? I said, there is this nice API that can tell me departures. I actually threw all of that into a database, and for one very easy reason, always start by narrowing down the options you have. If you take all trains departing at the station in Germany throughout a year, no model will be able to tell you that without extra information. What do you want to take? You want to take all departing trains on a day after a specific time because you know all the other choices are invalid. You now have means to tell your model, this is already your restricted search space. Additionally, we just add the departure time and the planned path into the prompt. The planned path, you will see it, it's basically just a pipe delimited list of the stations to which this train will go. Amazing data structure, but we're not API designers, we're the platform folks. It's not my API, so it's not my fault. Then you'll just say, find the next train to the destination. We do the exact same as we did before. We basically just put the departures into an integer, which is between 0 and the length minus 1 to choose from. We tell it to actually choose the departure. What it will do is it will indeed choose a departure. The departure is not here. Can I copy? Good that there is PowerPoint notes. That one cannot copy and put into Cursor. No module name, find next train, because it's an integer, find next train. What it will do is it will tell me, there is a next train leaving this station. It's going to Lindau-Reutin. It actually figured out the right planned path. It's departing 19 minutes past 10:00, which, yes, might be correct. If you arrive at 6:24 in Ulm and the model tells you that you can leave in 4 hours, and that would actually be truth, you would be mad. Rule 6: Narrow Down Deterministically Which actually brings me to my next points. The narrowing down we covered, but more importantly, don't let AI do the deterministic comparison. Instead of asking for the next train that departs here, ask for all trains that depart to Lindau, and then just check what is the train that is closest to my departure time using code. It's just very important to separate the AI logic and the code in this case, or you throw an MCP into it. You can build the MCP in Cursor. It takes probably 10 times the time of writing this one function. I would just write the function. Look at the amazing things we just got for free. We got easy metrics. We can go to the ops folks and tell them like, this is the accuracy of the model. I somewhat got ground truth down the road. This is how often the model failed the validation and I had to reiterate to ask it the question again. I can easily integrate with legacy because I now get an ID back and I can compare that ID to the database, or I get times back, which are structured as times. I know it is a time I can compare to the time that I have in the database. On the next level, I have a composable design. I can build around this like, ok, I have this one agent, which is just taking care of extracting the information from the screenshot. Then I have the next agent that maps it to my EVA numbers. Then this next agent, which gives me the departure time. That as well ties into, we now have a separated model view controller again. We have a state we can work with. We know what transitions we have to take to change the state. We have a separate mean to show it on the frontend side. The model is pretty straightforward. We basically have the input that we originally got, which is the screenshot. I need to hold this in the state. We have the output that we want to generate. In the end, I want to have the EVA number where I change the train and I want to have a number of the departure of the next train. I have this controller that just looks at the current state. It looks at where I want to get, and it just asks the LLM to do this one niche transition. Finally, and that's the coolest thing, we can satisfy business because the view can still be an AI chat interface, which is all they care about. It doesn't matter how it looks underneath as long as it looks like a ChatGPT interface. It is accepted by business. Conclusion There's of course a price we paid. We now have rigidity again. You need to define the state. If you want to change something in your process, you actually need to touch code again. You can't just like, I want this to be done as well. Just throw it in the prompt and it should work out. At the same time, this state that we got just pays off so much, because in exchange, we don't have to fight with the folks that are running signals. We don't really have to fight with our customers when we told them, yes, just go to this demolished train station somewhere in nowhere and look at the timetable. We could just eventually make everyone happy with the extra effort we put in. We force product to again think about what they actually want to achieve. They have to tell us what the outcome is. They can't be like, yes, just give me a bot that gives me the next departure. It's fine. That's enough requirement for you to build it. Always split up the things you do. Stick to best practices on your output schema. Next time you're stuck on a train, just figure out what you want to prompt today. There is always something you can prompt, except if you're between Munich and Bregenz because the internet connection just sucks. Questions and Answers Participant 1: You started out in your solution, leaving everything to the LLM and then didn't work. The end result that you arrived at, the part that the LLM played, am I correct in saying that the only thing that the LLM really did by itself, where you didn't really know how it happened, was to extract the station names from the screenshot and everything else was deterministic, just lookups of station names and stuff through JSON APIs? Jendrik Jordening: No. The lookup of the station name is as well LLM based, because I basically say like, I have all these probably misspelled station names and I just want to get the semantically closest one. It makes sense to do that with an LLM as well. Of course, the checking what kinds of trains depart after a certain time is deterministic, but then choosing the train that goes to Lindau is again LLM based. Literally everything that really has to do with text is given to the LLM because it's good at doing things that contain text. It just sucks at everything that has to do with numbers or should always do the same thing. Participant 1: Didn't you say at the end that when you let the LLM pick the next train, it took the one that's 4 hours, so your solution is to say, give me all the connections, which is an API call, and then the user makes a choice? Jendrik Jordening: No. You have the departures. I can show you how it looks in the XML. Let's just run this again. What you can see here is the departures are still somewhat unstructured. A departure of course always has a departure time, but it has this planned path, which just has the different stations. You still need an AI to figure out which train actually goes to where I want to go, but it should pick all the trains that go to the location you want to go to. Then the comparison between my current time and the departure time, you should of course do deterministically again. Really figuring out which train you can take is something you can better do with AI than doing like fuzzy matching on the planned paths. Participant 2: First of all, it's really comforting to know that other people are actually looking to help others that are suffering with Deutsche Bahn. I think it's interesting that you went with the approach of verifying stations. Have you also thought about taking into account the train lines that are actually going? For example, my use case is from Nuremberg to Munich and there are at least two lines. I think in summer there are also three lines that you use, but usually it's just two lines, and you know the stations along the lines. Have you thought about adding the flexibility to having multiple ways for the AI to figure out this information and piece it together? Jendrik Jordening: No, I didn't consider that. Actually, I went through Memmingen. I actually took another switch on the way because it was faster, but that was more on the DB side of how to solve it. That's why I was like, for actual choosing, it doesn't add value. That's why I left it out. Basically, you could figure that out as well if you add more planned paths with multiple hops. Participant 3: For the data validation step, did you actually put the validation result back into the LLM to figure out if it can choose better? Jendrik Jordening: That would be something where you can then do somewhat an LLM enabled controller. If you have your current state, the state you want to get to, and the history and conversation, that's where you could then actually use the LLM to decide where do you want to go next. You try to figure out the EVA number, you validate it. It tells you, no, you're wrong. Then you just basically throw it into the LLM, and there it is then as well interesting to do a mix of LLM enabled decision of what to do next and rule-based decision. Because after you extracted a station, you always want to look up the EVA number, but from there you could ask the user, I found the number. Do you want to see next departures, or do you want to see if you can get a car instead at the rental place, because you don't want to take the train or a bike because you're faster then? That would be then the point where you could then go back to the user, the user can answer it. Then, from there, you can choose your next action. See more presentations with transcripts