The Two-Pass Grading System: How Context Changes the Score

A developer built a two-pass LLM grading pipeline that scores chat messages from 1 to 5 for technical value, using heuristic length-and-regex filtering to drop about 12% of redundant data before inference and a second pass to re-evaluate ambiguous grade-3 messages. The system, running a local LLM on an RTX 4050 with 6GB VRAM, achieved a mean grading confidence of 0.89 against human-verified rankings.

In the previous post https://dev.to/mayank dewangan 08/why-i-chose-gemma4b-over-mistral-7b-38n0 we covered the hardware constraints of the system, which is an RTX 4050 with 6GB VRAM and an Intel i7 processor, and why we are using a local LLM. Now we turn to the grading pipeline itself. The condition for correctly grading a message involves a response in a defined format as attached below. This ensures we have explainable reasoning behind the grading, and if a human feels the grading is wrong, it can be changed by the user. Currently, we are storing them in JSON format, because while building the MVP, the focus is on building the complete working pipeline rather than building one scalable working component which is not needed at the moment, and will be required later during the process. As discussed in the earlier article, we are performing batch message grading. Let's dive deeper into how it actually works. Although we are performing batch grading, each message is still being graded as an individual message. That is, even if we are sending messages in a batch, the LLM grades them like message 1, then reason, then grade, then moves to message 2, and so on. We are restricting it to use the context of the whole message and decide the grade for the messages. This article specifically answers these two questions: We will get the answers by the end of this article. It is a valid concern, as we have only talked about the constraints we are working with and how we are going to get the data, but not how we are handling the data. The answer to the question is absolutely not. We are not dumping it to the LLM. We are first performing heuristic based message dropping to remove the obvious noise in the data. By combining length and regex, we are able to reduce the LLM load and reduce unnecessary LLM calls. On the real run, we managed to get rid of about 12% of the redundant data, which is exactly what we require. It is not perfect, but false negatives will cost us dearly in this scenario, as we don't want any important message to be dropped off before even reaching the LLM grading pipeline, which cannot be recovered later in the pipeline. The leniency on the heuristic is also because the LLM is being used as the safety net, which will automatically drop the messages which have no correlation to the tech domain. Now the question arises: how are we making the LLM grade the messages in a way which will act as a safety net, but also ensure that we retain all the important messages? Here is where it gets interesting. Each message is being graded on a scale of 1 to 5, where 1 and 2 are noise, social, or logistics. A score of 3 means it may be useful but lacks enough detail, and scores of 4 and 5 contain clear technical value, such as a solution, useful resource, practical experience, or measured result. The useful resources here are mostly GitHub repos, YouTube links, articles, or even research papers. So, our scraper shall also consider their data as well, right? Yeah, definitely. But extracting their data is a separate topic, reserved for future talks. The focus here is to understand whether we are just using the LLM blindly, and why we are not using the context of the batch if we are using batch grading. Though the first question is answered partially, it will get clearer near the end of the article. The grading system was first tested on the test dataset, where we personally cross checked the ranking of the message given by the LLM against the grading by a human no RL, just verifying if the grading system is actually working . We managed to get a mean grading confidence of 0.89 on actual data. On further investigation and iteration, we realized that grade quality 3 is quite ambiguous, as it is only getting 3 because it lacks enough data to support its position higher on the grading quality scale. Now, this is where our two-pass system comes in. We specifically take out all the data points being graded as 3 after the first pass, manually give the full metadata of the messages before and after the quality 3 message, but this time, we manually verify via code that the surrounding message is used as context, and the target is again the quality 3 message, which is re-graded on the same 1 to 5 scale. Now, we are using the batch as context to grade one message, but it is not contained in the system prompt. Rather, it is handled via code, and then verified if the same message has been graded. You may now question: why not do this the first time? After we received success with our shared grading system, we did try that, and it turned out to be a failure. Using the batch as context and asking Gemma-4b to use the whole batch as context and grade each message of the same batch, without properly segregating the target data and context data, is a bit too much work for our small model, which turned out to be a failure in terms of the response it gave in the format, or sometimes even the wrong grading of the messages. We also found out something interesting while experimenting with the batch as context and batch as target. We learned where the domain leverage comes in. We are not making it generalized for all domains and messages. Rather, we are making this for actually useful groups and teams, where they frequently talk about tech related topics and rarely go off track. As verified by the real data, individual messages in the tech domain can clearly pass through the grading system without having them bundled in the surrounding context, for most of the data points. This prevents us not only from unnecessary exploitation of our context window, but also from dumping useless information to the LLM, avoiding extra latency, and using the data smartly and handling the LLM according to our needs. We actually got 14.1% of the data points being upgraded to quality 4, and the average grading score went up by 2.3% after the second pass. Only the messages above 4 get ingested into the database. You can set the threshold value according to you. Here, we prioritized tech conversation, so that's why we went with 4 or above. One may object that we can directly pass the context and that adding the second pass system is adding latency. To the first question: every time we ask the LLM to use the same batch as context and as a target point, it is not merely a grading task. Rather, it is a complex task where the LLM has to decide which target it has graded, and when to use that as context. It may even cause data leakage, as the LLM might only give a higher result because the message it is grading is in the context of what it is seeing, which can be a disaster for our knowledge graph. Imagine every single message is directly dumped into your knowledge graph. Now, instead of keeping track of our information, we have a huge pile of trash which will spit out garbage by making even worse relationships than we can imagine. Also, we have tested the approach and got wrong grading for the same message when following that approach. So, that's why the two-pass grading system was introduced. For the latency part, there are two factors we would like you to consider. First is the aim of the project. It was designed and expected to take comparatively longer time on the first run, but make sure no noise enters your knowledge graph. So, we agreed this trade-off is worth the time. Second, we also have various links which we have to get through to truly make it our knowledge graph. So, going for the links will take time too, but for now I can only say that at the end, we will have both the data, as this connection will be understood in the next post, where we will cover this topic. Are we just dumping the messages? No. We are using a very controlled way on batches, making sure to cut off the obvious noise, and most importantly making sure it does not overflow the context window. By a rough estimation of token consumption of input data, if it is more than the context window, we dynamically fall back to lower batch sizes. Are we just replacing the paid LLM with a local LLM? No. We are managing the system grading, verifying if we get our result in the desired format, and most importantly using the context in a controlled way, and where necessary. Why not use batch as context and target in a single pass? Because it increases unnecessary complexity to the task, chances of data leakage, and most importantly, the domain we are building in provides us with the leverage that most tech talks are complete on their own without having to rely on the context much. And where it is required, we use our second grader pipeline. The two-pass grading system was not a design preference. It was a response to a hard constraint. On a 6GB VRAM machine, asking a small model to use the entire batch as both context and target in a single pass introduced more failure modes than it solved: schema violations, incorrect grading, and the risk of data leakage. By separating the tasks, first pass for individual grading and second pass for context aware re-grading of only the ambiguous cases, we achieved both reliability and resource efficiency. The lesson is simple. Not every problem should be solved in one pass. Sometimes, the smarter engineering choice is to split the work, handle each part with the right tool, and accept a small latency cost in exchange for cleaner data and a more trustworthy knowledge graph. Have you faced a similar trade-off between latency and data quality in your own pipelines? Did you choose a single pass approach and work around its limitations, or did you split the work into stages like we did? What's your approach to handling ambiguous cases in automated grading or classification systems? And if you've built a knowledge graph from conversational data, how did you handle noise and context? Drop your thoughts in the comments below. I read every response, and I'm especially curious about cases where a different architecture won out for you. If you've written about your own pipeline design, share a link. I'd love to learn from your approach. Let's make this a conversation, not just a case study.