When working at the Good Enough Agents workshop, I wanted a more practical way to talk about tokens and tokenizing.
The main idea is that LLMs don’t work with text; they work with tokens. We know this, but it is important to be specific: LLMs don’t work directly with tokens as text chunks, but with numbers associated with those chunks. I’ll call them token IDs here.
OpenAI published its tokenizing algorithm, Tiktoken, and I found a gem called tiktoken_ruby that wraps Tiktoken.
require 'tiktoken_ruby'
enc = Tiktoken.encoding_for_model('gpt-4o')
token_ids = enc.encode('Ruby is great!')
token_ids.map { |id| enc.decode([id]) }
enc.decode(token_ids)
Let’s try that on code too:
author_token_ids = enc.encode('class Author < ApplicationRecord')
session_token_ids = enc.encode('class Session < ApplicationRecord')
You can notice that the token ID arrays have the same length and only one element differs.
Now that you have the text as an array, you can do operations with it.
Let’s try array difference:
enc.decode(author_token_ids - session_token_ids)
enc.decode(session_token_ids - author_token_ids)
And then here is array intersection:
enc.decode(author_token_ids.intersection(session_token_ids))
I like this because it makes the tokens visible and shows a glimpse of how operations on an array of numbers representing text chunks could return something meaningful. Of course, this is not an LLM, just some array operations.
These integers don’t mean anything on their own. They are just IDs associated with those text chunks in a vocabulary.
You can see this by switching models and noticing it returns different numbers.
Tiktoken.encoding_for_model('gpt-4o').encode('class Author < ApplicationRecord')
Tiktoken.encoding_for_model('gpt-3.5-turbo').encode('class Author < ApplicationRecord')
This Tiktoken algorithm is specific to OpenAI. Claude or other LLMs might use a different algorithm. So again, these numbers don’t represent anything outside the algorithm they were produced with.