Can LLMs actually grade their own homework to get better results?

A new method called Consistency-based Self-adaptive Prompting (COSP) enables large language models to generate and filter their own few-shot examples, improving reliability without human-written prompts. The approach uses majority voting across multiple sampled responses to identify high-confidence pseudo-labels, which are then used as in-context examples for remaining queries. This reduces variance and removes the prompt-tuning bottleneck in production AI workflows.

Can LLMs actually grade their own homework to get better results? Consistency-based Self-adaptive Prompting COSP basically solves this by turning the LLM into its own data annotator. Instead of you writing the examples, the model generates them, and then uses a consistency check to figure out which ones are actually worth keeping. The logic behind the loop The core insight here is that variance in LLM outputs isn't just "noise"—it's a signal. If you ask a model a complex reasoning question five times and it gives you the same answer four times, it's likely correct. If it gives you five different answers, it's guessing. COSP leverages this by treating the model's own consistent answers as "pseudo-labels." It effectively creates its own few-shot prompt on the fly without a human ever having to write a single example. You get the guidance of few-shot prompting with the zero-effort setup of zero-shot. A real-world AI workflow for COSP If you want to implement this logic into your own LLM agent or pipeline, it follows a specific two-stage process: 1. The Generation Phase: You feed the model a batch of queries. For each query, you generate multiple independent responses sampling with a higher temperature . 2. The Filtering Phase: You look for consistency. Any response that matches the majority vote for that specific query is flagged as "high confidence." 3. The Adaptive Prompting Phase: You take those high-confidence pairs the original question and the consistent answer and feed them back into the prompt as in-context examples for the remaining unsolved queries. 4. Final Vote: The model generates a final set of answers based on these self-generated examples, and you take the majority vote again. Performance Breakdown Comparing this to standard methods, the shift in reliability is noticeable, especially in reasoning-heavy tasks: Zero-shot: Low effort, but high variance. The model often wanders off-track. Manual Few-shot: High accuracy if examples are perfect, but extremely brittle and doesn't scale across different task types. COSP: Low effort automated and higher stability. It dynamically adapts the examples to the specific distribution of the current dataset. For anyone building a production AI workflow, this is a huge win because it removes the "prompt tuning" bottleneck. You stop guessing which examples the model likes and let the model tell you what it finds consistent. It's a practical tutorial in self-correction that makes LLMs feel significantly more robust. Next crewai-go v0.4. → /en/threads/6728/ a guide to making money with AI https://tanyan888.com/ , with plenty of directly applicable cases.