{"slug": "representation-affects-retrieval-a-case-study-of-skill-discovery-and-routing-in", "title": "Representation Affects Retrieval: A Case Study of Skill Discovery and Routing in a Multimodal Agent Harness", "summary": "A case study of Tinycloud, a production multimodal video agent harness, reveals that in-prompt exposure of skills is not monotonically helpful: full autoload selects the gold skill on every task, while the production default misroutes one task due to lexical competition with an autoloaded tool-skill. The six-task ablation across three exposure regimes shows that all-off slows execution and produces hard discovery failures, highlighting the need for careful skill representation in agent harnesses.", "body_md": "arXiv:2608.20389v1 Announce Type: new\nAbstract: A production agent harness must discover and rank, from a growing library of skills, the one most appropriate for a user's task. At small scale this selection happens in context: the LLM planner chooses among skill representations exposed in its system prompt, without an explicit embedding-based retrieval step. We treat this in-context selection as the small-N counterpart to embedding-based skill retrieval at scale, and present a case study of how Tinycloud, a production multimodal video agent harness, represents its skills for the planner. The harness ships skills under two recurring representations: tool-skills that wrap a single external API or system tool and serve as primitive vocabulary, and workflow-skills that orchestrate tool-skill calls plus a template render to produce one named deliverable. The harness exposes them via two surfaces in the system prompt: an inlined-body surface (full instructions, scripts, templates) for autoloaded skills, and a one-line listing for on-demand skills. A six-task selection ablation across three exposure regimes (all-on, default, all-off) shows that full autoload selects the gold skill on every task; all-off slows execution and produces hard discovery failures; and the production default misroutes one task because its lexical signal collides with an autoloaded tool-skill that pulls planner attention away from a listed workflow-skill. The headline finding is that in-prompt exposure of skills is not monotonically helpful: partial exposure can create lexical competition that suppresses correct selection. We connect this small-N observation to recent retrieval-based skill-routing work at large scale, and frame this contribution as a case study rather than a benchmark.", "url": "https://wpnews.pro/news/representation-affects-retrieval-a-case-study-of-skill-discovery-and-routing-in", "canonical_source": "https://arxiv.org/abs/2608.20389", "published_at": "2026-08-24 04:00:00+00:00", "updated_at": "2026-08-24 04:13:09.785721+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-agents", "machine-learning"], "entities": ["Tinycloud", "arXiv"], "alternates": {"html": "https://wpnews.pro/news/representation-affects-retrieval-a-case-study-of-skill-discovery-and-routing-in", "markdown": "https://wpnews.pro/news/representation-affects-retrieval-a-case-study-of-skill-discovery-and-routing-in.md", "text": "https://wpnews.pro/news/representation-affects-retrieval-a-case-study-of-skill-discovery-and-routing-in.txt", "jsonld": "https://wpnews.pro/news/representation-affects-retrieval-a-case-study-of-skill-discovery-and-routing-in.jsonld"}}