cd /news/artificial-intelligence/evaluating-generative-agents-with-ac… · home topics artificial-intelligence article
[ARTICLE · art-48880] src=arxiv.org ↗ pub= topic=artificial-intelligence verified=true sentiment=· neutral

Evaluating Generative Agents with Actions Grounded in Socially Distributed Task Environments using Incognita

Researchers introduced Incognita, a framework for evaluating generative agents in socially distributed task environments where knowledge is partitioned across role-isolated participants. Testing on 18 retail tasks showed that stronger models improved success rates from 0% to 17.2% and reduced premature finalization from 100% to 58%, but overall reliability remained low.

read1 min views1 publishedJul 7, 2026

arXiv:2607.02975v1 Announce Type: new Abstract: Effective agency in social environments depends on when an agent seeks knowledge, when it acts, and whether its actions are justified by acquired information. Existing grounded benchmarks provide executable actions, persistent state, and verifiable outcomes, while social simulation environments provide rich interaction among language agents. We study an evaluation setting that combines these requirements. We define socially distributed task environments as interactive environments where task-relevant knowledge is partitioned across role-isolated participants and consequential actions are accessible only through them. Communication serves as exploration over role-partitioned knowledge, while grounded action serves as exploitation over environment state. We introduce Incognita, a Concordia-based framework that separates social interaction from grounded execution. The evaluated agent routes messages to a user or specialist entities; specialists mediate admissible operations; a deterministic sub-environment executes accepted operations over a canonical state; and an offline evaluator scores outcomes with inherited rewards. Incognita-Retail transforms tau-bench retail into a multi-entity environment while preserving final-state reward semantics. We evaluate three generative agent models on 18 tasks stratified by social breadth, with 540 trials. Progress appears in reward and behavior: success rises from 0 percent to 8.9 percent and 17.2 percent, while premature finalization falls from 100 percent to 87 percent and 58 percent. Stronger models elicit more hidden knowledge, contact more entities, and attempt more grounded writes, yet reliability remains low. These findings show that socially distributed task environments expose behavior before reliable success, including knowledge elicitation, source selection, grounded action attempts, and premature completion belief.

── more in #artificial-intelligence 4 stories · sorted by recency
── more on @incognita 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
Live at https://your-agent.zahid.host
Get free account → Pricing
from €0/mo · no card required
LIVE [news/evaluating-generativ…] indexed:0 read:1min 2026-07-07 ·