Tokens are an important foundational concept that underlie how we interact with LLMs. @greggyb and myself recorded a quick video to demystify what a token actually is and unpack several other related concepts.
#
What's in the video
- What is a token? Tokens are the basic unit of input and output that LLMs operate on.
- What is retrieval-augmented generation (AKA RAG)? Providing a model with extra data that isn't already in the model's training dataset, and is relevant to a specific query.
- What are context windows? Context windows are how many tokens a model can use for input and output. Context window size varies from model to model, but it has increased a lot in recent years.
- What is tokenmaxxing? Does using more tokens really mean you're being more productive? (Spoiler alert: no.)
To paint a bigger picture: more context you give a large language model, the better your responses are likely to be. The design patterns for providing that context have evolved at a rapid pace, and are likely to continue doing so.
What AI terms should we demystify next? Let us know!
source & further reading
dev.to — original article
Why Sick Patients Hate Your Cheerful Conversational Voice AI
I gave open claw and codex the whole internet without any api keys using this tool and it was never performed better
llms.txt: What It Actually Does, and Why It Rots