&. on X: "working with Astra to make a font where each token has equal width (early prototype) maybe this will make it easier to understand why models have the quirks they do"
working with Astra to make a font where each token has equal width (early prototype) maybe this will make it easier to understand why models have the quirks they do
working with Astra to make a font where each token has equal width (early prototype) maybe this will make it easier to understand why models have the quirks they do
I did not expect this post to get as much attention as it did... Anyway! With this webpage, you can make your own token-space font by combining a base font with a tokenizer of your choice
ampdot.mesh.host/token-space-fo… Disclaimer: GLM-5 & LLaMa 3 are still very unreliable. The other listed ones are accurate 99%+ of the time but there are still rare edge cases around merge rules, text wrapping, Unicode support, and kerning.
If you care about those things, please come back tomorrow The compiler page has instructions on how to install the font into Discord and Slack, so you can read AI outputs the way the AI wrote it (@poolside Laguna and @Arceeai Trinity are merely 97% on the subset of Project Gutenberg books used to test, so they also need fixes- come
OpenType has a 65,536 ligature limit and tokenizers are fucked up so this had to be implemented using GSUB and cmap The Claude one uses dynamic-programming and shortest-paths search inside the font to search for the most compact possible representation inside the tokenizer