AIArticle
How VS Code's under-the-hood optimizations and smart developer habits can slash your GitHub AI Credit consumption.
The transition of GitHub Copilot to usage-based billing on June 1, 2026, turned token efficiency from an academic optimization into a direct development expense. Under the new GitHub AI Credits model, every input, output, and cached token counts toward your bill.
This billing shift highlights a growing architectural tension. As developer tools transition from simple autocompletion to multi-turn agentic workflows, token consumption is skyrocketing. Agentic sessions run loops, call tools, and carry massive context windows. If you leave these agents to run unoptimized, they will quickly drain your credit balance and introduce severe latency.
To counter this, Microsoft and GitHub have rolled out a series of structural updates to the Copilot harness in VS Code. But platform-level updates only go so far. To truly stop the credit bleed, developers must understand how these optimizations work and adapt their daily coding habits accordingly.
The Anatomy of the Agentic Token Drain #
Every agentic request carries two primary token costs: the prompt prefix and tool-definition overhead.
The prompt prefix is the repeated foundation of a multi-turn conversation. It contains system instructions, active repository context, and the growing log of your conversation history. Because this prefix is sent with every single turn, it represents a massive chunk of your token budget.
Tool-definition overhead is the second major culprit. To let an agent interact with your environment, the runtime must explain what tools are available. Historically, this meant sending the name, description, and full JSON parameter schema for every single tool on every single turn. If you have an Model Context Protocol (MCP) server with 40 tools connected, that alone can inject 10 to 15 KB of schema overhead into every request. Even if that data is cached, it permanently eats into the model's active context window.
flowchart TD
A[Agentic Request] --> B[Prompt Prefix]
A --> C[Tool Definitions]
B --> D[System Instructions]
B --> E[Conversation History]
C --> F[JSON Parameter Schemas]
C --> G[Lightweight Metadata]
To mitigate these costs, VS Code and GitHub have introduced two core architectural changes: extended prompt caching and deferred tool search.
Under the Hood: Extended Caching and Tool Search #
Prompt caching allows the inference provider to reuse the computed model state (the key/value tensors) of a shared prefix instead of recomputing it from scratch. This is a massive win because cached input tokens can be up to 10 times cheaper than uncached ones, while also slashing time-to-first-token latency.
However, standard prompt caches are volatile. They typically live in fast GPU memory and expire after 5 to 10 minutes of inactivity. If you to think, write some code, or grab a coffee, your cache is wiped. Your next request becomes a costly cold start.
To solve this for OpenAI models, VS Code implemented extended prompt caching by passing the prompt_cache_retention: "24h"
body parameter. This moves the cached state to roomier, GPU-local storage, keeping it warm for up to 24 hours. The real-world impact of this change is stark. According to Microsoft's internal measurements, when requests are spaced 40 to 60 minutes apart, the relative cache hit rate for GPT-5.4 increased by 919%.
For tool overhead, VS Code now uses Tool Search to load schemas on demand. Instead of sending heavy JSON schemas upfront, the harness sends only lightweight metadata (names and descriptions). The model uses a defer_
flag (available in GPT-5.4 and newer) to request the full parameter schema only when it actually decides to call a specific tool. Because these deferred schemas are appended to the end of the context window rather than the prefix, they do not invalidate the cached prompt prefix.
The Developer Angle: Habits that Protect Your Wallet #
While these platform-level optimizations run automatically, their efficiency is highly dependent on how you interact with the IDE. If you write prompts poorly or manage your workspace inefficiently, you will bypass these guardrails.
1. Protect the Cache Boundary
The easiest way to blow your token budget is to invalidate your prompt cache mid-session. The prompt prefix remains cacheable only as long as it remains identical.
Avoid mid-session model or reasoning changes: Switching models or changing the reasoning effort level in the middle of a task forces the harness to discard the cache and reprocess the entire history under the new configuration.Start fresh conversations: A long, rambling thread carries its entire history into every new turn. When you finish a task and move to an unrelated problem, do not keep typing in the same window. Use/new
or/clear
in the CLI, or open a new chat session. This drops the accumulated context and starts a clean, cheap cache prefix.Compact long sessions: If you must keep a long session going, run the/compact
command to summarize the history and shrink the active context window.
2. Prune Your MCP and Extension Footprint
Do not treat MCP servers and developer extensions as "set and forget" utilities. Even with deferred tool search, having dozens of unused tools connected forces the agent to evaluate more metadata, which can lead to unnecessary exploration, incorrect tool calls, and wasted tokens.
Disable experimental extensions, one-off integrations, and unused MCP servers when they are not relevant to your current workflow. If you are in an implementation phase, you do not need your deployment or heavy research tools active.
3. Write Deterministic CI/CD Workflows
If you run automated agentic workflows in your CI/CD pipelines, token costs can accumulate silently and rapidly. You can optimize these workflows by replacing expensive, reasoning-heavy MCP tool calls with deterministic CLI commands.
Instead of letting an agent use an MCP tool to fetch a pull request diff (which requires an LLM turn to formulate the tool call, execute it, and process the response), use the GitHub CLI in a pre-agentic setup step:
gh pr diff > pr_diff.txt
Write this output to a workspace file and instruct the agent to read it directly. This eliminates an entire LLM round-trip, saves thousands of tokens, and lets the agent leverage its native training in file processing rather than tool execution.
4. Leverage Auto Model Selection
Default to Copilot's auto model selection. This routing layer analyzes your prompt and sends it to the cheapest model capable of handling the task, reserving expensive reasoning models for complex architectural debugging.
As an added incentive, using auto model selection grants a 10% discount on model costs. Crucially, the router is designed to protect your cache: it will only switch models at natural cache boundaries, such as the start of a new session, ensuring you do not accidentally trigger a costly mid-session cache invalidation.
The Verdict #
Token efficiency is no longer just a concern for the engineers building LLM infrastructure; it is now a core discipline for the developers using them. VS Code's implementation of extended prompt caching and deferred tool provides a powerful foundation, but it requires active cooperation from the user. By treating your agentic sessions as ephemeral, keeping your tool footprint lean, and off data fetching to deterministic CLI tools, you can dramatically lower your latency and keep your GitHub AI Credit consumption under control.
Sources & further reading #
Improving token efficiency for GitHub Copilot in VS Code— code.visualstudio.com - Optimizing your AI usage to maximize efficiency and reduce cost - GitHub Docs— docs.github.com - How to Lower GitHub Copilot Token Cost: User Habits from VS Code's Internal Optimizations - SmartScope— smartscope.blog - Improving token efficiency in GitHub Agentic Workflows - The GitHub Blog— github.blog
Rachel Goldstein· Dev Tools Editor
Rachel has been embedded in the developer tooling ecosystem for nearly eight years, covering everything from IDE wars and package-manager drama to the quiet rise of AI-assisted coding. She has a soft spot for open-source maintainers and an unhealthy number of terminal emulators installed on a single laptop.
Discussion 0 #
No comments yet
Be the first to weigh in.