{"slug": "explaining-attention-with-program-synthesis", "title": "Explaining Attention with Program Synthesis", "summary": "Researchers at an undisclosed institution developed a method to replace attention heads in transformer language models with human-readable Python programs, achieving over 75% similarity in attention patterns on TinyStories and incurring only a 16% average perplexity increase when replacing 25% of heads in GPT-2, TinyLlama-1.1B, and Llama-3B. The approach uses program synthesis to generate executable code that reproduces attention behavior, advancing interpretability in deep learning.", "body_md": "# Computer Science > Machine Learning\n\n[Submitted on 17 Jun 2026]\n\n# Title:Explaining Attention with Program Synthesis\n\n[View PDF](/pdf/2606.19317)\n\n[HTML (experimental)](https://arxiv.org/html/2606.19317v1)\n\nAbstract:A longstanding goal of research on interpretable deep learning is to replace opaque neural computations with human-meaningful symbolic descriptions. In this paper, we propose an approach for approximating the behavior of components of deep networks with executable programs. We focus on attention heads in transformer language models. For a given head, we first compute its associated attention matrices on a collection of randomly selected training examples. Next, we prompt a pre-trained language model with a summary of these matrices, and instruct it to generate a set of Python programs that can reproduce the associated attention patterns given only text from the input sentence. Finally, we re-rank programs according to how well our final set of programs predict behavior on held-out inputs. We demonstrate that a set of fewer than 1,000 such generated programs can reproduce the attention patterns of heads in GPT-2, TinyLlama-1.1B, and Llama-3B, achieving an average Intersection-over-Union similarity above 75% on TinyStories. Moreover, the best-fit programs can replace neural attention heads without substantially affecting model behavior: replacing 25% of attention heads with programmatic surrogates across the three models incurs only a 16% average perplexity increase, while maintaining performance on a variety of downstream question answering benchmarks. This work contributes a scalable pipeline for reverse-engineering attention heads in transformer models using human-readable, executable code, advancing a path toward symbolic transparency in neural models.\n\n### References & Citations\n\nLoading...\n\n# Bibliographic and Citation Tools\n\nBibliographic Explorer\n\n*(*[What is the Explorer?](https://info.arxiv.org/labs/showcase.html#arxiv-bibliographic-explorer))\nConnected Papers\n\n*(*[What is Connected Papers?](https://www.connectedpapers.com/about))\nLitmaps\n\n*(*[What is Litmaps?](https://www.litmaps.co/))\nscite Smart Citations\n\n*(*[What are Smart Citations?](https://www.scite.ai/))# Code, Data and Media Associated with this Article\n\nalphaXiv\n\n*(*[What is alphaXiv?](https://alphaxiv.org/))\nCatalyzeX Code Finder for Papers\n\n*(*[What is CatalyzeX?](https://www.catalyzex.com))\nDagsHub\n\n*(*[What is DagsHub?](https://dagshub.com/))\nGotit.pub\n\n*(*[What is GotitPub?](http://gotit.pub/faq))\nHugging Face\n\n*(*[What is Huggingface?](https://huggingface.co/huggingface))\nScienceCast\n\n*(*[What is ScienceCast?](https://sciencecast.org/welcome))# Demos\n\n# Recommenders and Search Tools\n\nInfluence Flower\n\n*(*[What are Influence Flowers?](https://influencemap.cmlab.dev/))\nCORE Recommender\n\n*(*[What is CORE?](https://core.ac.uk/services/recommender))\nIArxiv Recommender\n\n*(*[What is IArxiv?](https://iarxiv.org/about))# arXivLabs: experimental projects with community collaborators\n\narXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.\n\nBoth individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.\n\nHave an idea for a project that will add value for arXiv's community? [ Learn more about arXivLabs](https://info.arxiv.org/labs/index.html).", "url": "https://wpnews.pro/news/explaining-attention-with-program-synthesis", "canonical_source": "https://arxiv.org/abs/2606.19317", "published_at": "2026-06-18 06:34:32+00:00", "updated_at": "2026-06-18 06:52:53.001322+00:00", "lang": "en", "topics": ["machine-learning", "large-language-models", "ai-research", "ai-safety"], "entities": ["GPT-2", "TinyLlama-1.1B", "Llama-3B", "TinyStories"], "alternates": {"html": "https://wpnews.pro/news/explaining-attention-with-program-synthesis", "markdown": "https://wpnews.pro/news/explaining-attention-with-program-synthesis.md", "text": "https://wpnews.pro/news/explaining-attention-with-program-synthesis.txt", "jsonld": "https://wpnews.pro/news/explaining-attention-with-program-synthesis.jsonld"}}