{"slug": "evaluator-optimizer", "title": "Evaluator Optimizer", "summary": "A technical walkthrough details the Evaluator-Optimizer workflow, in which one LLM call generates a response and a second LLM call evaluates it and returns feedback in a loop until the output passes. The pattern is recommended when clear evaluation criteria exist and iterative refinement adds value, and the example implementation loops generate and evaluate calls until the evaluator returns \"PASS\", demonstrated on a task to implement a Stack with push(x), pop(), and getMin() in O(1) time.", "body_md": "## Evaluator-Optimizer Workflow\n\nEvaluator-Optimizer Workflow\n\nIn this workflow, one LLM call generates a response while another provides evaluation and feedback in a loop.\n\n### When to use this workflow\n\nWhen to use this workflow\n\nThis workflow is particularly effective when we have:\n\n- Clear evaluation criteria\n- Value from iterative refinement\n\nThe two signs of good fit are:\n\n- LLM responses can be demonstrably improved when feedback is provided\n- The LLM can provide meaningful feedback itself\n\nfrom util import extract_xml, llm_call\n\ndef generate(prompt: str, task: str, context: str = \"\") -> tuple[str, str]:\n\n    \"\"\"Generate and improve a solution based on feedback.\"\"\"\n\n    full_prompt = f\"{prompt}\\n{context}\\nTask: {task}\" if context else f\"{prompt}\\nTask: {task}\"\n\n    response = llm_call(full_prompt)\n\n    thoughts = extract_xml(response, \"thoughts\")\n\n    result = extract_xml(response, \"response\")\n\n    print(\"\\n=== GENERATION START ===\")\n\n    print(f\"Thoughts:\\n{thoughts}\\n\")\n\n    print(f\"Generated:\\n{result}\")\n\n    print(\"=== GENERATION END ===\\n\")\n\n    return thoughts, result\n\ndef evaluate(prompt: str, content: str, task: str) -> tuple[str, str]:\n\n    \"\"\"Evaluate if a solution meets requirements.\"\"\"\n\n    full_prompt = f\"{prompt}\\nOriginal task: {task}\\nContent to evaluate: {content}\"\n\n    response = llm_call(full_prompt)\n\n    evaluation = extract_xml(response, \"evaluation\")\n\n    feedback = extract_xml(response, \"feedback\")\n\n    print(\"=== EVALUATION START ===\")\n\n    print(f\"Status: {evaluation}\")\n\n    print(f\"Feedback: {feedback}\")\n\n    print(\"=== EVALUATION END ===\\n\")\n\n    return evaluation, feedback\n\ndef loop(task: str, evaluator_prompt: str, generator_prompt: str) -> tuple[str, list[dict]]:\n\n    \"\"\"Keep generating and evaluating until requirements are met.\"\"\"\n\n    memory = []\n\n    chain_of_thought = []\n\n    thoughts, result = generate(generator_prompt, task)\n\n    memory.append(result)\n\n    chain_of_thought.append({\"thoughts\": thoughts, \"result\": result})\n\n    while True:\n\n        evaluation, feedback = evaluate(evaluator_prompt, result, task)\n\n        if evaluation == \"PASS\":\n\n            return result, chain_of_thought\n\n        context = \"\\n\".join(\n\n            [\"Previous attempts:\", *[f\"- {m}\" for m in memory], f\"\\nFeedback: {feedback}\"]\n\n        )\n\n        thoughts, result = generate(generator_prompt, task, context)\n\n        memory.append(result)\n\n        chain_of_thought.append({\"thoughts\": thoughts, \"result\": result})\n\n### Example Use Case: Iterative coding loop\n\nExample Use Case: Iterative coding loop\n\nevaluator_prompt = \"\"\"\n\nEvaluate this following code implementation for:\n\n1. code correctness\n\n2. time complexity\n\n3. style and best practices\n\nYou should be evaluating only and not attemping to solve the task.\n\nOnly output \"PASS\" if all criteria are met and you have no further suggestions for improvements.\n\nOutput your evaluation concisely in the following format.\n\n<evaluation>PASS, NEEDS_IMPROVEMENT, or FAIL</evaluation>\n\n<feedback>\n\nWhat needs improvement and why.\n\n</feedback>\n\n\"\"\"\n\ngenerator_prompt = \"\"\"\n\nYour goal is to complete the task based on <user input>. If there are feedback\n\nfrom your previous generations, you should reflect on them to improve your solution\n\nOutput your answer concisely in the following format:\n\n<thoughts>\n\n[Your understanding of the task and feedback and how you plan to improve]\n\n</thoughts>\n\n<response>\n\n[Your code implementation here]\n\n</response>\n\n\"\"\"\n\ntask = \"\"\"\n\n<user input>\n\nImplement a Stack with:\n\n1. push(x)\n\n2. pop()\n\n3. getMin()\n\nAll operations should be O(1).\n\n</user input>\n\n\"\"\"\n\nloop(task, evaluator_prompt, generator_prompt)\n\n```\n=== GENERATION START ===\nThoughts:\n\nThe task requires implementing a Stack with constant time operations including finding minimum. \nTo achieve O(1) for getMin(), we need to maintain a second stack that keeps track of minimums.\nEach time we push, if the value is smaller than current min, we add it to minStack.\nWhen we pop, if the popped value equals current min, we also pop from minStack.\n\nGenerated:\n\n``` python\nclass MinStack:\n    def __init__(self):\n        self.stack = []\n        self.minStack = []\n        \n    def push(self, x: int) -> None:\n        self.stack.append(x)\n        if not self.minStack or x <= self.minStack[-1]:\n            self.minStack.append(x)\n            \n    def pop(self) -> None:\n        if not self.stack:\n            return\n        if self.stack[-1] == self.minStack[-1]:\n            self.minStack.pop()\n        self.stack.pop()\n        \n    def getMin(self) -> int:\n        if not self.minStack:\n            return None\n        return self.minStack[-1]\n```\n\n=== GENERATION END ===\n\n=== EVALUATION START ===\nStatus: NEEDS_IMPROVEMENT\nFeedback: \nWhile the implementation is generally correct and achieves O(1) time complexity for all operations, there are several areas for improvement:\n\n1. Error Handling:\n- pop() should raise an exception when stack is empty rather than silently returning\n- getMin() should raise an exception when stack is empty rather than returning None\n- These behaviors should be consistent with standard stack implementations\n\n2. Type Hints:\n- Return type hint for pop() should be None or void\n- Missing type hints for class variables stack and minStack\n\n3. Documentation:\n- Missing docstrings for class and methods\n- Missing parameter descriptions\n\n4. Edge Cases:\n- No explicit handling of invalid input types for push()\n\nThe core algorithm is correct and efficient, but the implementation could be more robust and better documented for production use.\n\n=== EVALUATION END ===\n\n=== GENERATION START ===\nThoughts:\n\nBased on the feedback, I'll improve the implementation by:\n1. Adding proper error handling with exceptions\n2. Including type hints and docstrings\n3. Adding input validation\n4. Maintaining O(1) time complexity for all operations\n\nGenerated:\n\n``` python\nfrom typing import List\n\nclass MinStack:\n    \"\"\"A stack that supports push, pop, and getting minimum element in O(1) time.\"\"\"\n    \n    def __init__(self):\n        \"\"\"Initialize empty stack with two internal lists.\"\"\"\n        self.stack: List[int] = []\n        self.minStack: List[int] = []\n        \n    def push(self, x: int) -> None:\n        \"\"\"\n        Push element onto stack and update minimum stack.\n        \n        Args:\n            x: Integer to push onto stack\n            \n        Raises:\n            TypeError: If x is not an integer\n        \"\"\"\n        if not isinstance(x, int):\n            raise TypeError(\"Input must be an integer\")\n            \n        self.stack.append(x)\n        if not self.minStack or x <= self.minStack[-1]:\n            self.minStack.append(x)\n            \n    def pop(self) -> None:\n        \"\"\"\n        Remove and return top element from stack.\n        \n        Raises:\n            IndexError: If stack is empty\n        \"\"\"\n        if not self.stack:\n            raise IndexError(\"Cannot pop from empty stack\")\n            \n        if self.stack[-1] == self.minStack[-1]:\n            self.minStack.pop()\n        self.stack.pop()\n        \n    def getMin(self) -> int:\n        \"\"\"\n        Get minimum element in stack.\n        \n        Returns:\n            int: Minimum element in stack\n            \n        Raises:\n            IndexError: If stack is empty\n        \"\"\"\n        if not self.minStack:\n            raise IndexError(\"Cannot get minimum from empty stack\")\n        return self.minStack[-1]\n```\n\n=== GENERATION END ===\n```\n\n", "url": "https://wpnews.pro/news/evaluator-optimizer", "canonical_source": "https://platform.claude.com/cookbook/patterns-agents-evaluator-optimizer", "published_at": "2026-09-23 15:18:10+00:00", "updated_at": "2026-09-23 15:30:26.326684+00:00", "lang": "en", "topics": ["large-language-models", "ai-agents", "ai-tools", "generative-ai"], "entities": [], "alternates": {"html": "https://wpnews.pro/news/evaluator-optimizer", "markdown": "https://wpnews.pro/news/evaluator-optimizer.md", "text": "https://wpnews.pro/news/evaluator-optimizer.txt", "jsonld": "https://wpnews.pro/news/evaluator-optimizer.jsonld"}}