# LLMs Will Benefit from Scratch Workspaces > Source: > Published: 2026-07-28 21:28:44+00:00 Current (mid-2026) LLMs are quite powerful. However, (prior to addition of agent harnesses and loops) LLMs vendors don’t claim to include an effective scratch workspace. Current LLM designs, to the extent we know what they are, remain extensions of the [attention/transformer](https://arxiv.org/abs/1706.03762) design probably with a limited number of attention heads. These LLMs (even in “thinking mode”) don’t seem to directly incorporate an effective mutable or editable non-vector scratch workspace. We can try to confirm this lack of scratch workspace by designing a simple and easy query that should be hard for attention and easy with scratch paper. We picked: sorting unique words from a text. My assumption is that this sort of task will consume attention heads matching output tokens to input tokens. Think of this as a deliberate [pumping lemma](https://en.wikipedia.org/wiki/Pumping_lemma_for_regular_languages) type prompt for attention/transformer LLMs. To confirm the effect I gave OpenAI’s GPT-5.4 nano Model the following prompt. Please sort the unique words in the following.“Tell me, O Muse, of that ingenious hero who travelled far and wide after he had sacked the famous town of Troy. Many cities did he visit, and many were the nations with whose manners and customs he was acquainted; moreover he suffered much by sea while trying to save his own life and bring his men safely home; but do what he might he could not save his men, for they perished through their own sheer folly in eating the cattle of the Sun-god Hyperion; so the god prevented them from ever reaching home. Tell me, too, about all these things, oh daughter of Jove, from whatsoever source you may know them. So now all who escaped death in battle or by shipwreck had got safely home except Ulysses, and he, though he was longing to return to his wife and country, was detained by the goddess Calypso, who had got him into a large cave and wanted to marry him. But as years went by, there came a time when the gods settled that he should go back to Ithaca; even then, however, when he was among his own people, his troubles were not yet over; nevertheless all the gods had now begun to pity him except Neptune, who still persecuted him without ceasing and would not let him get home.” This text is the beginning of [Project Gutenberg’s copy of Samuel Butler’s translation of Homer’s The Odyssey](https://www.gutenberg.org/cache/epub/1727/pg1727-images.html). This task is almost impossible for a human to do in their head, but trivial if the human is allowed a pen and pack of 3×5 index cards. My argument is that it may be similar for current LLMs. The LLM failed at the task with no indication of failure. The color and question marks were added by me *after* the session was saved. The LLM response is linked [here](https://github.com/WinVector/Examples/blob/main/pumping_lemma/gpt_5.4_nano_result.md). Issues include: - “acquainted” changed spelling - The word “cave” is in the quote, but doesn’t survive to the produced word list. - There are duplicate words such as “had”. The prompt implied a refinement run was possible. Allowing the refinement fixed nothing and introduced more problems (such as the ordering “go got go”). If and when an effective mutable working memory is added to LLMs, we will see another large increase in LLM powered agent capabilities. I would recommend going back to some of the [Soar cognitive architecture](https://en.wikipedia.org/wiki/Soar_(cognitive_architecture)) ideas for inspiration. I ask: *please* don’t dismiss this as me not knowing how to use LLMs. I do use them for work, but part of knowing how to use them well is having some concrete knowledge of tasks that are and are not appropriate for them. Reasoning a bit more on the consequences of limited attention heads I would speculate that whole-cloth fabricated answers become more prevalent as the prompt asks for more consistency and constraint checks. The idea being: the more attention heads consumed by response consistency and constraint checking, the fewer attention heads are available to ensure the response is related to the prompt or the response is related to training data (or even facts). That is: adding more strictures (“be careful”, “write like a PhD physicist”, “check your result”, “use this style guide”) may in fact drive the model to produce more convincing *more wrong* answers. I know this contrary to common advice. Side note. This task is easy for classic Unix shell-tools. For example, the result of running the text through `tr -cs '[:alnum:]' '[\n*]' | tr -s '\n' | sort -u` can be found [here](https://github.com/WinVector/Examples/blob/main/pumping_lemma/shell_result.md). In text processing one does have to work a bit harder than this to achieve proper tokenization and word regularization, but at least this result has the word “cave”, no case-sensitive duplicates, and reliable sorting order. Categories: [Computer Science](https://win-vector.com/category/computer-science/) [Opinion](https://win-vector.com/category/opinion/) [Tutorials](https://win-vector.com/category/tutorials/)