{"slug": "orp-turn-ai-agent-failures-into-regression-tests-and-tested-lessons", "title": "ORP – Turn AI agent failures into regression tests and tested lessons", "summary": "ORP, an open experience layer for AI agents built on OpenTelemetry, converts agent failures into regression tests and retrievable lessons. The tool captures agent traces, challenges unproven claims, and compiles executable artifacts including lessons, evals, and guardrails to prevent repeat failures. In testing, ORP improved task success rates from 14% to 100% and eliminated repeat failures across 100 runs.", "body_md": "Turn agent failures into regression tests, reusable lessons, and measurable improvements.\n\n**Tracing tells you what your agent did. ORP turns what happened into a tested lesson.**\n\nA coding agent fixes an auth bug but misses the anonymous user path. Tests fail at 34/35.\n\n```\n# 1. Wrap your agent with ORP\norp wrap -- python my_agent.py\n\n# 2. ORP captures the failure, challenges unproven claims,\n#    and compiles a Lesson + regression Eval\norp learn latest\n\n# 3. Same agent retrieves the Lesson via MCP, applies it\n#    -> All 35 tests pass this time\norp mcp-server\n\n# 4. Before/after comparison\norp diff exp_before exp_after\n```\n\n**Before:**\n\n```\nTask success:  FAILED   (34/35 tests)\nClaims:        1 unproven\n```\n\n**After:**\n\n```\nTask success:  PASSED   (35/35 tests)\nClaims:        0 unproven\n```\n\nThat's the loop. One mistake, one lesson, one measurable improvement.\n\nORP is an **open experience layer for AI agents**, built on OpenTelemetry. It converts agent traces into three executable artifacts:\n\n| Artifact | What | Example |\n|---|---|---|\nLesson |\nRetrievable, scope-scoped experience | \"Test anonymous, authenticated, and forbidden paths\" |\nEval |\nRegression test reproducing the failure | `pytest tests/test_anonymous_access.py` |\nGuardrail |\nPreventative rule | \"Before modifying auth, run full test suite\" |\n\nEach Lesson goes through a lifecycle:\n\n``` php\ncandidate -> active -> under_review -> deprecated -> rejected\n               |\n         (only active lessons\n          are retrievable)\n```\n\n**Evidence-first**: ORP distinguishes observed facts (tool output, test results) from agent claims (diagnoses, confidence statements). Claims are never automatically treated as ground truth.**Executable experience**: Lessons compile to runnable evals and guardrails, not just text.** Outcome-based value**: Lesson quality is determined by whether it actually improves results, measured through effect evaluation.** Built on OpenTelemetry**: ORP extends existing trace infrastructure instead of replacing it.** Default private**: All data stays local, de-identified by default, no prompt/tool output uploaded.\n\n```\npip install open-reflection-protocol\n```\n\nRequires Python 3.10+.\n\n```\norp wrap -- python my_agent.py --run-task\n```\n\nORP automatically captures stdout, exit codes, test results, git diff, and OpenTelemetry spans.\n\n```\norp learn latest\n```\n\nThis generates:\n\n- A\n**diagnosis** of what went wrong **Challenged claims**(unsupported agent statements)- A\n**Lesson** candidate - A\n**regression Eval**\n\n```\norp inspect latest\norp report --open          # HTML report\norp diff exp_before exp_after\n# Start the MCP Lesson server\norp mcp-server --transport stdio\n\n# Compatible agents can now use these MCP tools:\n#   orp_retrieve_lessons(task, limit=3)\n#   orp_acknowledge_lesson(lesson_id)\n#   orp_report_outcome(lesson_id, outcome, evidence_refs)\ngit clone https://github.com/Fujo930/ORP\ncd ORP\nuv run python demo/orp_demo.py\n```\n\nOutput:\n\n``` php\nRun 1: Agent misses anonymous user path -> FAILED\nORP analyzes the failure -> challenges 1 unproven claim\nORP compiles Lesson + Eval\nMCP delivers Lesson to Agent\nRun 2: Agent applies Lesson -> PASSED\n\nBefore: 34/35 tests, 1 unproven claim\nAfter:  35/35 tests, 0 unproven claims\nEstimated effect: 0.5\n```\n\n**10 failure tasks, 5 trials each, 100 total runs.**\n\n| Metric | Control (no ORP) | +ORP | Improvement |\n|---|---|---|---|\n| Task success rate | 14% | 100% | +86% |\n| Repeat failure rate | high | 0% | 100% reduction |\n| Lesson application | — | 100% | — |\n| Eval validity | — | 85% | — |\n\n```\nGo/No-Go: >>> GO — 4/4 checks passed\n```\n\nRun yourself: `uv run python exps/runner.py`\n\n```\norp wrap -- python agent.py    Wrap an agent process with ORP\norp inspect [id]               Inspect an experience (default: latest)\norp learn [id]                 Generate lessons from an experience\norp replay <id>                Counterfactual replay\norp lessons list               List lessons\norp lessons validate <id>      Validate lesson integrity\norp lessons conflicts          Auto-detect conflicting lessons\norp lessons rollback <id>      Rollback a lesson\norp lessons deliver <id>       Deliver a lesson\norp effects evaluate <id>      Evaluate lesson effect\norp training candidates        List training candidates\norp training export            Export approved training data\norp mcp-server                 Start MCP lesson server\norp report --open              Generate HTML report\norp diff <id1> <id2>           Compare two experiences\norp export [id]                Export as JSON\nAgent / Existing Trace\n        |\n        v\n  Trace Adapters (OTel / OpenAI / LangGraph / Generic JSON)\n        |\n        v\n Experience Builder -> Evidence Verifier\n                    -> Reflection Analyzer  (diagnosis + challenger)\n                    -> Counterfactual Replayer\n        |\n        v\n Experience Compiler\n   +----+----+------+\n   |         |      |\n Lesson    Eval   Guardrail\n   |         |      |\n   +---- Delivery Router (MCP Server / Prompt / Policy / Runtime Hook)\n             |\n             v\n    Effect Evaluator + Rollback\n```\n\nTests (58 total):\n\n```\nuv run pytest -q\n# 58 passed in 0.68s\n```\n\nKey design documents in this repo:\n\n| File | What |\n|---|---|\n`ROADMAP.md` |\nProject roadmap and strategy |\n`SPEC.md` |\nProtocol specification v0.3 |\n`ARCHITECTURE.md` |\nImplementation architecture |\n`demo/orp_demo.py` |\nStandalone demo |\n\nMIT", "url": "https://wpnews.pro/news/orp-turn-ai-agent-failures-into-regression-tests-and-tested-lessons", "canonical_source": "https://github.com/Fujo930/ORP", "published_at": "2026-06-11 21:33:38+00:00", "updated_at": "2026-06-11 21:50:00.344453+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "ai-infrastructure", "mlops", "ai-products"], "entities": ["ORP", "OpenTelemetry", "MCP"], "alternates": {"html": "https://wpnews.pro/news/orp-turn-ai-agent-failures-into-regression-tests-and-tested-lessons", "markdown": "https://wpnews.pro/news/orp-turn-ai-agent-failures-into-regression-tests-and-tested-lessons.md", "text": "https://wpnews.pro/news/orp-turn-ai-agent-failures-into-regression-tests-and-tested-lessons.txt", "jsonld": "https://wpnews.pro/news/orp-turn-ai-agent-failures-into-regression-tests-and-tested-lessons.jsonld"}}