A new theoretical paper asks how much an autoregressive transformer retains of its prompt after generating t tokens, and reports a conditional no-hair theorem: when two different prompts are followed by the same continuation text, the distance d between their hidden states shrinks as generation proceeds. The work frames prompt influence as decaying with the number of generated tokens, so that differences attributable to the prompt alone diminish over the course of a continuation.
We ask how much an autoregressive transformer retains of its prompt after generating t tokens. Consider two different prompts followed by the same continuation text, so that any difference in their hidden states comes from the prompt alone, and we find that the distance between their hidden states d
EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.