This is a submission for DEV's Summer Bug Smash: Clear the Lineup powered by Sentry.
I've been working on a Tokyo public transit MCP server for a while (see the previous posts in the series). Recently, I started experimenting with adding a SKILL.md
to see whether giving the agent more structured guidance would improve its behavior and output.
Instead of treating this as simply another feature, I turned the project into a small debugging and experimentation exercise of How MCP tools and Skills work together, and adding another layer of guidance actually makes an agent better (or worse).
At a high level, MCP provides the agent with capabilities and tools, while a Skill provides task-specific guidance for when and how those capabilities should be used.
The goal was simple: produce more accurate, consistent, and useful real-time transit answers.
The MCP tools themselves worked correctly, but introducing the Skill layer caused unexpected changes in agent behavior. In some cases, the agent even started preferring web search over the MCP tools.
Nothing crashed down to the ground, but the problem was a silent change in the agent's decision path.
I tested the same transit requests With Skill and Without Skill, then compared the outputs.
I looked at tool usage, route selection, transfers, travel time, fares, and overall output quality across 6 progressively more complex evaluations.
| Eval | Query Type | With Skill vs Without Skill | Current Finding |
|---|---|---|---|
| 1 | Simple station-to-station route | Nearly identical | Little measurable Skill impact |
| 2 | Station → attraction route | Very similar | MCP already handles the task well |
| 3 | Station → landmark route | Identical | Little measurable Skill impact |
| 4 | Natural-language, more complex route request | Different routes | Skill changed behavior, but selected a less optimal route |
| 5 | Route request with explicit transfer preference | Different routes | Skill better aligned the route with the user's preference |
| 6 | Time-sensitive route planning | Identical | MCP handled the task well; Skill had no measurable impact |
The results challenged my initial assumption that adding a Skill layer would automatically produce better or more optimized outputs.
For simple tasks, Eval 1-3 showed little difference between With and Without Skill. The MCP tools were already capable of handling these requests well. Eval 4 was the first surprise. The Skill changed the route selection, but the result was actually less optimal.
Eval 5 showed the opposite. When the user explicitly asked to minimize transfers, the Skill helped align the agent's route selection with that preference.
Eval 6 added another interesting result. For time-sensitive arrival planning, both versions produced the same result using the same MCP capability. In this case, the Skill did not provide a measurable benefit.
The good news is that the MCP tools were already capable of handling station-to-station routing, attraction and landmark routing, and time-sensitive arrival planning without an additional guidance layer.
Adding guidance is not automatically an improvement. Its value depends on the task, the underlying tools, and how the Skill itself is defined.
More instructions don't necessarily mean better agent behavior. Sometimes the best improvement might be knowing when not to add another layer.
This experiment is not finished yet.
Next, I want to expand the eval suite with more complex requests that combine time constraints, transfer preferences, and fare limitations. The goal is to identify where the Skill can actually improve route optimization rather than simply change agent behavior.
Once I understand where the Skill adds value, I'll refine SKILL.md
and rerun the evaluations against the baseline.
I also plan to introduce Google ADK-based evaluation and tracing to look deeper into tool selection and agent behavior beyond final-output comparison.
The "Bug" wasn't a broken MCP tool. The tools were working.
The unexpected behavior came from adding another layer of guidance and changing the agent's decision path.
That turned out to be a more interesting problem to investigate than simply fixing a broken tool.
In agent systems, adding more intelligence doesn't necessarily mean getting better results. Sometimes the real challenge is making sure the layers work together.
Transparency Note: I used AI as an editor, not as the author. For this article, it helped refine the structure and improve the English grammar. The technical content, experiments, opinions, and conclusions are my own and were reviewed by me before publishing.