{"slug": "agent-memory-benchmarks-need-more-than-an-overall-score", "title": "Agent memory benchmarks need more than an overall score", "summary": "The Agent Memory Challenge (AML) Cycle 2 results drew 402 task submissions across text, code and multimodal memory tracks and generated more than 1 million cross-platform impressions and over 10 million cumulative website clicks, according to the @AgentMemoryL leaderboard organizers. Khipu, a system that stores every message verbatim and converts photos to text for search, took 1st place on the Multimodal Open-Source leaderboard with 43.26 points, while Gedo Memory placed 3rd on the Text Memory Commercial Products board and 5th among closed-source methods. Analysis from @omarsar0 showed capability breakdowns that overall rankings can hide: on the closed-source text board the top 10 systems scored between 70 and 78 on Memory Governance, and Personalization & Care was the lowest-scoring category.", "body_md": "🏆 AML Cycle 2 results are live! 💥\n• 1M+ cross-platform impressions\n• 10M+ cumulative website clicks\n• 402 task submissions\n• Three tracks: text, code and multimodal memory\nThis cycle brought together teams from across the global AI community, including teams fromShow more\n\nA closer look at AML Cycle 2 from @omarsar0: capability breakdowns reveal differences that overall rankings can hide. Personalization & Care stands out as an area for further investigation.\nThanks for the thoughtful analysis—we’ll share more detailed breakdowns and cases soon!\n\nI like how this measures different aspects of agent memory capabilities.\nThis is part of the @AgentMemoryL Cycle 2 results.\nOn the closed-source text board, the top 10 systems score between 70 and 78 on Memory Governance.\nPersonalization & Care is the lowest-scoring category\n\nKhipu took 1st place in the AML Cycle 2 Multimodal Open-Source leaderboard with 43.26 points! 🥇\nCongratulations to @Lv999APTITUDE, and thank you for contributing to the new track. Excited to see what comes next for the project!\n\nKhipu: memory for AI agents. Stores every message verbatim, never rewrites it; photos become text for search and come back as evidence.\n1st in Multimodal · Open-source Methods, Agent Memory Challenge Cycle 2.\n@AgentMemoryL#AgentMemoryLeaderboardgithub.com/Lv999APTITUDE/…\n\nCongratulations to @gedolabs on Gedo Memory’s 3rd-place finish in the AML Cycle 2 Text Memory Commercial Products leaderboard! 🥉\nGreat to see the team sharing lessons from the evaluation and putting them into practice. Looking forward to what comes next👏\n\nGedo Memory（gedo.ai） placed 3rd on the AML Cycle 2 Commercial Products leaderboard (Text Memory), and 5th among closed-source methods. I'm grateful to our team, and to the AML and CSIG organizers.\nA few notes on what this meant for us.\nWhy we entered\nAt Gedo", "url": "https://wpnews.pro/news/agent-memory-benchmarks-need-more-than-an-overall-score", "canonical_source": "https://x.com/AgentMemoryL/", "published_at": "2026-11-20 09:17:31+00:00", "updated_at": "2026-11-20 09:44:37.435783+00:00", "lang": "en", "topics": ["ai-agents", "ai-research", "artificial-intelligence"], "entities": ["Agent Memory Challenge", "Khipu", "Gedo Memory", "@omarsar0", "@AgentMemoryL", "@Lv999APTITUDE", "CSIG"], "also_reported_by": [], "takeaway": "Overall leaderboard ranks can mask capability gaps: on AML Cycle 2's closed-source text board the top 10 systems clustered between 70 and 78 on Memory Governance, while Personalization & Care was the weakest category.", "recommendation": {"advice": "Break agent memory evals into per-capability categories like Memory Governance and Personalization & Care instead of relying on one overall score, since top-10 systems clustered within 8 points on governance.", "playbook": "evals-testing", "playbook_url": "https://wpnews.pro/playbooks/evals-testing"}, "alternates": {"html": "https://wpnews.pro/news/agent-memory-benchmarks-need-more-than-an-overall-score", "markdown": "https://wpnews.pro/news/agent-memory-benchmarks-need-more-than-an-overall-score.md", "text": "https://wpnews.pro/news/agent-memory-benchmarks-need-more-than-an-overall-score.txt", "jsonld": "https://wpnews.pro/news/agent-memory-benchmarks-need-more-than-an-overall-score.jsonld"}}