{"slug": "tokens-token-ids-and-array-operations-in-ruby", "title": "Tokens, token IDs and array operations in Ruby", "summary": "A Ruby developer demonstrates how OpenAI's Tiktoken tokenizer, wrapped in the tiktoken_ruby gem, converts text into token IDs and enables array operations like difference and intersection to reveal meaningful text chunks, such as extracting 'Author' or 'Session' from Ruby class definitions. The post emphasizes that token IDs are model-specific and meaningless outside their algorithm, as shown by different IDs for the same text across GPT-4o and GPT-3.5-turbo.", "body_md": "When working at the [Good Enough Agents workshop](https://goodenoughagents.com), I wanted a more practical way to talk about tokens and tokenizing.\n\nThe main idea is that LLMs don’t work with text; they work with tokens. We know this, but it is important to be specific: LLMs don’t work directly with tokens as text chunks, but with numbers associated with those chunks. I’ll call them token IDs here.\n\n[OpenAI](https://openai.com) published its tokenizing algorithm, [Tiktoken](https://github.com/openai/tiktoken), and I found a gem called [tiktoken_ruby](https://rubygems.org/gems/tiktoken_ruby) that wraps Tiktoken.\n\n```\nrequire 'tiktoken_ruby'\nenc = Tiktoken.encoding_for_model('gpt-4o')\n\ntoken_ids = enc.encode('Ruby is great!')\n# => [134047, 382, 2212, 0]\n\ntoken_ids.map { |id| enc.decode([id]) }\n# => [\"Ruby\", \" is\", \" great\", \"!\"]\n\nenc.decode(token_ids)\n# => \"Ruby is great!\"\n```\n\nLet’s try that on code too:\n\n``` js\nauthor_token_ids  = enc.encode('class Author < ApplicationRecord')\n# => [1444, 10764, 464, 12493, 6721]\n\nsession_token_ids = enc.encode('class Session < ApplicationRecord')\n# => [1444, 17681, 464, 12493, 6721]\n```\n\nYou can notice that the token ID arrays have the same length and only one element differs.\n\nNow that you have the text as an array, you can do operations with it.\n\nLet’s try array difference:\n\n``` js\nenc.decode(author_token_ids - session_token_ids)\n# => \" Author\"\n\nenc.decode(session_token_ids - author_token_ids)\n# => \" Session\"\n```\n\nAnd then here is array intersection:\n\n``` js\nenc.decode(author_token_ids.intersection(session_token_ids))\n# => \"class < ApplicationRecord\"\n```\n\nI like this because it makes the tokens visible and shows a glimpse of how operations on an array of numbers representing text chunks could return something meaningful. Of course, this is not an LLM, just some array operations.\n\nThese integers don’t mean anything on their own. They are just IDs associated with those text chunks in a vocabulary.\n\nYou can see this by switching models and noticing it returns different numbers.\n\n```\nTiktoken.encoding_for_model('gpt-4o').encode('class Author < ApplicationRecord')\n# => [1444, 10764, 464, 12493, 6721]\n\nTiktoken.encoding_for_model('gpt-3.5-turbo').encode('class Author < ApplicationRecord')\n# => [1058, 7030, 366, 55926]\n```\n\nThis Tiktoken algorithm is specific to OpenAI. Claude or other LLMs might use a different algorithm. So again, these numbers don’t represent anything outside the algorithm they were produced with.", "url": "https://wpnews.pro/news/tokens-token-ids-and-array-operations-in-ruby", "canonical_source": "https://allaboutcoding.ghinda.com/tokens-token-ids-array-operations-ruby/", "published_at": "2026-09-09 09:13:10+00:00", "updated_at": "2026-09-09 10:13:36.363925+00:00", "lang": "en", "topics": ["large-language-models", "developer-tools"], "entities": ["OpenAI", "Tiktoken", "tiktoken_ruby", "Ruby", "GPT-4o", "GPT-3.5-turbo"], "alternates": {"html": "https://wpnews.pro/news/tokens-token-ids-and-array-operations-in-ruby", "markdown": "https://wpnews.pro/news/tokens-token-ids-and-array-operations-in-ruby.md", "text": "https://wpnews.pro/news/tokens-token-ids-and-array-operations-in-ruby.txt", "jsonld": "https://wpnews.pro/news/tokens-token-ids-and-array-operations-in-ruby.jsonld"}}