cd /news/artificial-intelligence/capturing-in-context-learning-dynami… · home › topics › artificial-intelligence › article
[ARTICLE · art-173078] src=machinebrief.com ↗ pub= topic=artificial-intelligence verified=true sentiment=↑ positive

Capturing In-Context Learning Dynamics with Task Operators

A paper posted to arXiv (2610.01054v2) introduces Task Operator (TO), a method that captures in-context learning as an analytically derived update to the attention output projection, letting language models replay a task's transformation without reprocessing the full demonstration set. The authors show each attention head's output is an affine transformation of its context-masked counterpart whose parameters are empirically stable across samples for a given task, and that TO achieves the best overall performance among prior methods across lexical, algorithmic and reasoning tasks, substantially narrowing the gap between zero-shot inference and ICL. The work also finds the extracted knowledge concentrates in a task-specific sparse circuit across layers and positions, and that averaging operators from disjoint demonstration batches enables many-shot scaling without expanding the context window; code is available at github.com/gzxiong/task_operator.

recommendation

Replace repeated full-demonstration ICL passes with a Task Operator update to the attention output projection, and average operators from disjoint demonstration batches to scale to many shots without growing the context window.

more in playbook: Inference and self-hosting
by read1 min views2 publishedNov 20, 2026

arXiv:2610.01054v2 Announce Type: replace Abstract: In-context learning (ICL) enables language models to perform new tasks from demonstrations without weight updates. However, every ICL inference requires processing the full set of examples, resulting in inefficient deployments, and how ICL works mechanistically is not fully understood. Prior work compresses ICL into fixed activation vectors extracted from specific layers or positions, but these input-independent interventions fail on complex tasks where the output depends on fine-grained interactions with the input. By analyzing the ICL forward pass, we show that each attention head's output is an affine transformation of its context-masked counterpart, and that the parameters of this transformation are empirically stable across samples for a given task. Building on this, we introduce Task Operator (TO), which replays this transformation as an analytically derived update to the attention output projection. Across lexical, algorithmic, and reasoning tasks, TO achieves the best overall performance among prior methods and substantially narrows the gap between zero-shot inference and ICL. We further show that the extracted knowledge concentrates in a task-specific sparse circuit across layers and positions, and that averaging operators from disjoint demonstration batches enables effective many-shot scaling without expanding the context window. Our code is available at https://github.com/gzxiong/task_operator.

── more in #artificial-intelligence 4 stories · sorted by recency
── more on @task operator 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/capturing-in-context…] indexed:0 read:1min 2026-11-20 · —