How does an AI agent pick from 686 skills in a second?

The article describes an empirical test of a "semantic router" pattern for AI agents, where 686 skills from a community corpus were indexed using an embedding model to enable fast, sub-second skill retrieval. The test achieved 62.5% strict top-1 accuracy and 87.5% top-5 cluster accuracy, while using only ~500 tokens per task compared to the ~228K tokens required to load all skill names and descriptions into the system prompt—a 456x context-window saving. The author explains that this approach decouples the skill catalog from the prompt, solving the scalability and attention-quality issues of the default "progressive disclosure" method.

I ran an empirical test on the "skills as semantic router" pattern for Claude Code agents. I indexed 686 randomly sampled skills from a 4,556-skill community corpus into mesh-memory, embedded them with a single sentence-transformer model, and ran a fixed set of eight task queries through it. Here are the headline numbers: strict top-1 accuracy 62.5%, top-5 cluster accuracy 87.5%, sub-second query latency, ~500 tokens loaded per task versus the ~228K tokens just to keep names + descriptions of all 4,556 skills in the system prompt the default behavior, even with Anthropic's progressive disclosure . That is roughly a 456x context-window saving with the right skill landing in the agent's top-5 candidates seven times out of eight. This post explains why I ran the test, how it was set up, what the results actually show, and where the pattern breaks honestly. The full source for the runner and queries is reproducible. Anthropic's Claude Code skills and Cursor's equivalents, and every other agent framework's skills ship as markdown files in a folder. Each one has a name and a short description in its frontmatter. The default loading strategy is what Anthropic calls "progressive disclosure": the agent reads every skill's name + description into its system prompt at startup, and only loads the full body when it decides to invoke one. Progressive disclosure handles the body problem — you do not pay for skill bodies you never use. But it does not handle the index problem. Even just the names and descriptions are loaded for every skill, every session, before any work starts. At fifty skills you spend roughly 2.5K tokens on the catalog. At 200 skills the catalog eats 5% of a Claude Sonnet 200K context window before you have asked anything. The math gets ugly fast: Even when the catalog physically fits, attention quality on a long list of similar items degrades. Past about 1,000 entries the agent starts making wrong picks on hand-eye-distinguishable cases. There is also no garbage collection: nobody removes stale skills, nobody flags duplicates, the catalog only grows. The semantic router pattern decouples the catalog from the prompt. Each skill's name and description is stored once in an embedding index, with a tag pointing to the SKILL.md file on disk. At task time the agent runs a single semantic-search call against the task description, gets the top-5 candidates, picks one, and reads the full body only for the one it picked. Token cost per turn is constant regardless of catalog size. That is the theory. The question is whether the search actually returns relevant skills in real conditions. Corpus. antigravity-awesome-skills, a public collection of Anthropic-format skills from the community. 4,556 SKILL.md files, deduplicated by directory. Each file has a YAML frontmatter name, description, tags and a markdown body. Sample. 1,000 skills selected by random.shuffle seed=42 from the sorted file list. Of those, roughly 200 were silently rejected by the mesh bulk-ingest endpoint likely a content-validation filter , 25 failed in transit on a single wave, and 86 ended up stuck in the "pending" state after embedding — a known mesh-memory worker stall. Final indexed corpus: 686 skills. Routing document. For each skill, the embedded text is exactly name + "\n\n" + description . The full SKILL.md body stays on disk; mesh holds only the routing signal plus a skill path tag with the disk path. That is roughly 50-200 tokens per skill in the index. Embedding model. intfloat/multilingual-e5-base , running locally via sentence-transformers, 768-dimensional vectors, stored in Postgres + pgvector. Ten parallel embedding workers, throughput ~38 docs/minute on a single CPU container. Queries. Eight diverse task descriptions, written before looking at the corpus, designed to span common dev work: For each query I asked mesh for the top-5 most similar skills and inspected the names plus cosine similarity scores. Metrics. After all 686 skills were indexed, query-by-query: Strict top-1: 5 of 8 = 62.5%. Top-5 cluster: 7 of 8 = 87.5%. Cosine similarity range for top-1: 0.83-0.88. Query latency: under 1 second across all runs. The corpus was built in waves of 100 skills, with the full query suite re-run after each wave. That gives a sense of how router quality scales with corpus depth: one query timeout on the run Two observations land hard. Top-5 cluster saturates early. By 500 indexed skills about 11% of the full corpus the cluster metric was already at 85% and barely moved with the next 186. For most queries the relevant family of skills was already in the index; what changed later was which member led the cluster. Strict top-1 keeps climbing. Going from 500 to 686 indexed skills bumped top-1 from 50% to 62.5%. The improvement came from one specific skill xvary-stock-research finally landing in the sampled portion. Each new wave was a chance for an exact-match skill to surface for one of the eight queries. The SQL query never produced a real top-1. The top-1 was food-database-query a false match on the word "query" , and the cluster contained spark-optimization and cqrs-implementation but no actual SQL-tuning skill. Looking at the unindexed portion of the corpus, sql-optimization-patterns exists — it just landed in waves 10-45 of the shuffle, beyond our 1,000-sample window. This is the honest face of the pattern. Router accuracy is bounded by corpus depth, not by the search algorithm. Embeddings did their job perfectly: every query returned coherent clusters with similarities in the 0.83-0.88 range. When the right skill was in the index, the router found it. When it was not, no search quality compensates. The practical consequence: this pattern wins as your skill collection grows. With under 30 skills there is no point — eager-load them and move on. Past 100, the math starts mattering. Past 1,000, it is the only sustainable option. Per task turn: That is a 100x to 450x reduction depending on which skill ends up being read. The kicker: the router cost is constant whether the catalog holds 100 skills or 100,000. Stuck-on-embed leak. Of 772 documents that made it into the mesh database, 86 11.1% ended up stuck in the "pending" state and never got an embedding. The mesh-memory worker code has MAX CONSECUTIVE ERRORS = 10 , which halts a worker on a bad document and silently leaves the rest of the queue. Worth filing as an upstream issue; in the meantime, restarting the worker on a fresh container drains the backlog. Silent bulk-ingest drops. About 200 of 1,000 documents sent never appeared in the database at all — the bulk PUT returned success but mesh stored fewer rows than were posted. Likely a content-validation filter on empty or near-empty documents. Worth investigating but did not affect the search-quality results. The 10-worker bump. Default NUM WORKERS = 3 is bottlenecking on a single-CPU container. Bumping to 10 quadrupled throughput ~9 → ~38 docs/min with no observed degradation. This change should be parameterizable in the next mesh release. Query timeouts. One query in the wave-5 run timed out at the default 30s; rerunning succeeded. Likely a cold-cache warmup issue, not a search-correctness problem. Everything used in this test is open source. The runner is roughly 70 lines of Python: Drop the corpus into mesh, run a few queries against it, and look at what you get back. If the pattern fits your workflow, the wiring is trivial: a single MCP call in your claude.md that says "before working on any task, search mesh for type:skill matching the task description, then load the top result." If you have not read it yet, the companion post How to Organize Your Claude Skills Without Drowning in Files covers the versioning side of this same pattern — storing skills as memory documents with date tags, so the latest version is one query away and old versions never need to be deleted. Together the two posts cover both "how do you store skills sanely" and "how do you retrieve them sanely." Originally published on klymentiev.com