Do Coding Agents Need Skills? Researchers at Georgian's AI Lab, collaborating with Expo, analyzed telemetry from two internal hackathons in which developers built mobile apps with Claude Code to measure how coding agents use Anthropic's Skills. They found that agents often succeeded without invoking the relevant skill, relying instead on intrinsic ecosystem knowledge and MCP server tools, and that installing a skill with keyword-rich descriptions does not guarantee the agent will use it. The team concludes that skills should be designed around a different goal than simply supplying information agents can already find. Anthropic introduced Skills in late 2025 as a way of providing information to agents. Skills can be used for the dual purpose of either providing the right information to the agent to work in an environment or to provide the agent with a user’s preferences and biases. Since this introduction a number of platforms have started shipping their own version of skills which are designed to provide guidance to an agent on how to interact with the platform’s system. However, LLMs and agents are inherently stochastic in nature and often the amount of information needed to be conveyed across all capabilities on a platform is too large to be contained in a single skill. The information constraints skills face have led us to ask two related questions: 1. How well are agents using skills? 2. How do skills impact the outcome of the task they are designed to assist with? Members of Georgian’s AI Lab and I collaborated with Expo who ran two internal hackathons where we asked participants to switch on telemetry for their Claude Code usage as they built mobile applications. In this post we share the insights collected from our analysis of the telemetry data to understand how well agents understood the Expo ecosystem out-of-the-box. We examined how often agents reached for the skills at their disposal, how often they succeeded in their task with or without skills and whether skills made a difference to the final outcome. The results of our analysis suggest that agents have become adept at finding the right information by using the MCP servers provided in the plug-in and recovering from failure. Further, our research demonstrates that while agents do reach for skills, they may also use a variety of other tools to find the right information. Specifically, installing a skill and writing a description with the right keywords does not guarantee that an agent will use the skill. In our work with Expo it became clear that the agent had a good intrinsic understanding of Expo’s ecosystem as evidenced by its consistent, successful use of the ecosystem’s various CLIs without relying on the relevant skill. We found that agents use the tools available to them consistently well, which suggests that skills should be built around a different goal. What is the Expo plug-in? The Expo plug-in ships 21 skills and one MCP server. Most of the skills fall into three broad groups: the framework itself expo-router , expo-ui , expo-native-ui , expo-data-fetching , the build and release path eas-workflows , eas-app-stores , eas-hosting , eas-simulator , eas-observe , eas-update-insights , and migration or extension work expo-upgrade , expo-module , expo-brownfield , expo-web-to-native . The MCP server is a hosted endpoint at mcp.expo.dev and across our traces, we see it expose 12 tools split between looking up documentation search documentation , read documentation and reading live account state workflow , build , testflight . Of the 21 skills included in the plug-in, 15 appeared in our analysis and the remaining six were absent. The six absent skills govern work that nobody in our test set was engaged in, including: expo-project-structure which has no mechanical signature we can look for easily , expo-app-clip , expo-brownfield , expo-data-fetching , expo-examples and expo-web-to-native . It is important to understand how Claude Code loads skills into its context. Claude does not input the full contents of a skill at the start of a session, but rather puts the name and a one- or two-line description of every available skill as written by the skill creator into its context. The body of a skill only enters the context when the agent decides to invoke it, and the reference files that some of the larger skills carry are read separately, on demand, after that. Some harnesses would also truncate the content of the skill depending upon the size and the number of skills being loaded example, Codex though we didn’t do experiments to check this for Claude Code specifically. Finally, our participants were not all on the same version of the Expo plug-in. The skills we see loaded come from Expo plug-in versions 1.8.3, 1.8.4, 1.8.5, 1.9.0 and 1.9.4, and in one case from a local checkout of the skills repository. It is important to note that this doesn’t skew our analysis. What data did we collect? Agent telemetry usually comes in the form of traces. A trace from a single session of agent-human interaction consists of the history of all the user messages, the agent messages and all the tool calls the agent made which includes actions like executing bash commands, looking up information on the web and in the case of Expo, initiating a build on EAS, etc . This data is arranged temporally and provides a rich substrate for understanding the behavior and capability of the agent. From our hackathons, we received traces from 10 different users working on 12 different codebases across 131 different sessions. A user often worked on the same codebase across many different sessions, some of which even overlapped Imagine a user with multiple terminal windows open each running their own instance of Claude Code, collaborating on the same codebase. Across the 131 different sessions in our dataset we have 1,388 human turns, 38,846 agent turns, 20,065 tool calls, 85 hours of agent working time and 42M tokens. Hence, it’s a substantial amount of data from which we should be able to glean non-trivial insights. We share some characteristics of our trace dataset in Appendix A.1 as well as a list of apps people built. What did the agent do? In this section, we look at the agent’s work to see what steps it performed as it worked on the above codebases. Specifically, we look at the various tool calls it made and how frequently it reaches for Expo skills. As we will see, even though most of the sessions were specifically about building Expo applications and the Expo plug-in was available in 115 out of 131 sessions, only a very small fraction of tool calls come from the Expo plug-in in the form of its MCP server and the agent invokes an Expo skill from the 21 different skills present in the plug-in only 33 times out of 131 sessions. In the next section, we will look into how this impacts the actual work done by the agent. For now, we take a closer look at the actions of the agent to understand where the agent is spending most of its work. We start by looking at what tools and skills the agent invokes. This will give us a good bird’s-eye view of how and what the agent is trying to do. Then we zoom into the information-gathering aspect, as it is the most direct motivation for building skills. Specifically, we look at where the agent is getting its information from and what fraction of it is coming from skills. Lastly, we look at how the agent reacts to a user mentioning a skill name directly or indirectly in their prompt. Did tool and skill use change the outcome? The Claude Code harness which surrounds Anthropic’s models provides the models with a plethora of tools, these are actions that the agent can perform in the environment it sits in to accomplish the task it’s working on. It can include actions like executing bash commands, reading or writing a file, looking up information, etc. In Claude Code, even accessing a skill is a tool call and we see that the agent reaches for this tool a total of 86 times across all the sessions. Tools are typically defined by the harness builders and their choice is usually optimized with a combination of observed behavior from the models as well as what’s present in the models’ training data. For example, in a Claude Code harness there are two tools to write to a file, Edit and Write. The Edit tool replaces an exact string inside an existing file that the agent has already read, while the Write tool creates a new file or overwrites an existing file with the same name. Look at Figure 1 for the frequency of most invoked tools as well as MCP servers which are themselves a type of tool . Also look at Figure 2 for the frequency of most invoked skills. It is worth paying attention to what the two MCP servers with the largest number of invocations were doing as they substantially outweigh other MCP servers in the traffic they are getting. The first one is paper , a design canvas, with 1,847 calls, and the second one is argent , which drives a running app on a simulator, with 1,221 calls. Together they make for 95% of the 3,216 MCP calls, as opposed to the 110 for the Expo MCP. Both of these MCPs are used to close a visual loop for the agent. The paper server gives the design as the source of truth write html, get screenshot , update styles , while argent taps, swipes and takes a screenshot of the app to check what actually rendered gesture-tap , await-ui-element , debugger-reload-metro . We also found that participants pasted 123 images in their prompts across all the sessions. This gives a strong indication of the substantially UI-heavy work that the agent undertook in these sessions. It also gives us insight into how coding agents are being used for mobile app development where the agent is responsible not only for writing code but also for making sure that the app works and has a good UI by looking at the screen and interacting with it. As you can see in Figure 2, argent skills were some of the most invoked skills. Where is the agent getting its information? Skills are often designed to be a primary source of information for the agent. Yet an agent can get the needed information from multiple sources such as looking up documentation online, accessing documentation through the MCP server, looking at the help option of various CLIs or simply relying on its internal knowledge. As we are building the right set of skills to go with the Expo platform it is important to understand where the agent gets its information. In Figure 3 we look at this exactly. There are four sources of information considered in that plot: skills, MCP, web and accessory–an umbrella term for other more circuitous routes. A categorization of these is present in Appendix A.2. We also plot the number of sessions where the agent uses any two of the first three, for all three combinations these are the light-blue bars . There are many insights to glean from Figure 3. We see that in 50 of the 131 sessions, the agent used none of the four categories that we recognized. In looking at these 50 sessions more closely we see that 26 of these had no Expo relevant work in them. In the other 15 the agent traces have some mention of Expo but the agent did not fetch any Expo-specific information. The other 9 sessions were trivial sessions where the agent made less than 3 tool calls in total . Looking more closely at the second category we see that the sessions were mainly: 1. Iterating on UI inside an already-established codebase 9 sessions : The user asks for a visual change and the agent edits existing screens, copying the patterns already there: 1. “look at paper mcp. and implement the new components for th home screen… make sure to use styling im currently using ad copy the designs” → 10 edits to tabs /index.tsx , new progress-card.tsx , words-to-master.tsx 2. “look at the progressive blur on the tab bar. i want to put this at the top of the screen too” → tabs / layout.tsx 3. “add expo-haptics and add haptics on most of the buttons” → 8 edits each to app/new-pattern.tsx , app/tempo.tsx , app/lane-editor.tsx 4. “maybe add 7 more icons to round out at 50” → app/change-icon.tsx The APIs touched are expo-router , @expo/ui , expo-glass-effect , expo-haptics , NativeTabs . But the instruction is “copy the designs I’m currently using”, and the file already imports the API. Since the API usage is already present in the codebase, the agent feels confident in not having to look up that information again. 1. Conducting repo hygiene that happens to touch Expo files 4 sessions : This includes ESLint fixes, PR rebases, addressing code-review comments and changing a default sort order. Nothing that needs Expo specific knowledge. 2. Debugging live account state 2 sessions : This includes EAS CLI and read-only queries about things that had already happened 1. “why is it, on testflight when i try installing this app, its saying the requested app is not available” → eas build:view , eas whoami , eas submit:list --json , eas build:list . 2. Another session is the agent helping compose a reply to Apple Developer Support. So basically, in these 15 sessions the agent didn’t need extensive information about Expo. It could go off of internal knowledge or the code that it was looking at. Overall, what this demonstrates is that the agent looks up Expo-specific information whenever it’s actively working with Expo-specific concepts but there is a wide variety of ways in which the agent looks up this information, often using multiple modalities in the same session. Another striking part about Figure 3 is the difference between the “accessory” and “any of the above” bars. The accessory routes are within four sessions of every route combined. We present a detailed breakdown of the accessory routes in Table 2 in Appendix A.2. From this we see that these routes were deliberately invoked by the agent even though in 43% of the cases the information was present in some skill. This implies that the agent reaches for a wide variety of sources when it needs to find information and a skill might not always be the most obvious choice. Making this information available in just skills would probably not enable the agent to do its best work; the information needs to be present on every surface that the agent interacts with. How does the agent react to a user mentioning the skill? Ideally, in Claude Code, a user invokes a skill by using /