{"slug": "attention-is-simpler-than-you-think-a-hand-crafted-superhero-transformer", "title": "Attention is simpler than you think - A hand-crafted superhero transformer", "summary": "TechAarvam's workshop materials include a hand-crafted transformer example that demonstrates the intuition behind the attention block. The notebook uses a simple vocabulary and hand-coded attention weights to show how attention extracts information for a feedforward network, which is trained to predict corrections to word attributes.", "body_md": "*Part of the TechAarvam workshop support files — [Build Your Own Model](https://www.techaarvam.com/workshops/build-your-own-model).*\n\nThis notebook presents a hand-crafted example. The goal is to understand the intuition behind the attention block in the transformer architecture.\n\nAttention and FFN are the two main components. FFN is an Artificial neural network (ANN) with 1 input, 1 hidden, and 1 output layer.\n\nSo we start constructing the FFN's job by hand. Then we show how Attention extracts what the FFN can use from a sentence. The sentences used and the vocabulary are intentionally hand-designed to be simple. The attention block weights are hand-coded, instead of trained. The ANN(FFN) is trained.\n\n``` python\nimport numpy as np\n\nnp.set_printoptions(precision=2, suppress=True)\n```\n\n| Word | can fly? | has wheels? | can speak? | \n|---|---|---|---|\n| Rock | 0 | 0 | 0 | \n| Human | 0 | 0 | 1 | \n| Car | 0 | 1 | 0 | \n| Talking Tow Truck | 0 | 1 | 1 | \n| Crow | 1 | 0 | 0 | \n| Flying Superhero | 1 | 0 | 1 | \n| Plane | 1 | 1 | 0 | \n| Talking Planes | 1 | 1 | 1 | \n\n```\nword_to_attributes = {\n    \"Rock\":              (0, 0, 0),\n    \"Human\":             (0, 0, 1),\n    \"Car\":               (0, 1, 0),\n    \"Talking Tow Truck\": (0, 1, 1),\n    \"Crow\":              (1, 0, 0),\n    \"Flying Superhero\":  (1, 0, 1),\n    \"Plane\":             (1, 1, 0),\n    \"Talking Planes\":    (1, 1, 1),\n}\n\nfor word, bits in word_to_attributes.items():\n    print(f\"{word:20} {bits}\")\nRock                 (0, 0, 0)\nHuman                (0, 0, 1)\nCar                  (0, 1, 0)\nTalking Tow Truck    (0, 1, 1)\nCrow                 (1, 0, 0)\nFlying Superhero     (1, 0, 1)\nPlane                (1, 1, 0)\nTalking Planes       (1, 1, 1)\n```\n\nThe ANN's output - the logits (probability scores) for the next word is a correction to the input word.\n\nThe input is fixed bit-encoding of 6 bits.\n\n```\nInput word  - 3 bits, one-hot, or IDs (design choice)\nCorrection  - 3 bits\nOutput word - attributes after the correction\nattributes_to_word = {bits: word for word, bits in word_to_attributes.items()}\n\ndef apply_correction(word, correction):\n    \"\"\"Correction is 3 bits: 1 means flip that attribute.\"\"\"\n    bits = word_to_attributes[word]\n    out = tuple(b ^ c for b, c in zip(bits, correction))\n    return attributes_to_word[out]\n\nprint(apply_correction(\"Rock\", (1, 0, 0)))   # add flight\nprint(apply_correction(\"Human\", (1, 0, 0)))  # add flight\nCrow\nFlying Superhero\n# Pseudocode - not run here.\n#\n# for loop in range (num_epochs):\n#     for batch_inputs, batch_targets in loader:\n#         prediction = model.forward(batch_inputs)          # predict\n#         loss = CrossEntropyLoss(prediction, target)       # how far off?\n#         loss.backward()                                   # gradient descent\n```\n\nIn the full workshop, the notebook with the full ANN implementation is available. Visit the relevant TechAarvam pages to locate them.\n\nInput is now a sentence:\n\n**Crow keep-flight swap-speech he-is?**\n\nSentences have complex structure. Word order varies.\n\nSo: fixed sentence structure, small defined vocabulary.\n\n```\nObjects:       Rock, Human, Crow, Flying superhero\nActions:       swap-flight, swap-speech, keep-flight, keep-speech\nInterrogative: he-is?\n```\n\n8-bit vectors, shown as 4 + 4.\n\n**Bits 1-4 - attributes and correction:**\n\n| Token | can fly | can speak | swap flight | swap speech | \n|---|---|---|---|---|\n| Rock | 0 | 0 | 0 | 0 | \n| Human | 0 | 1 | 0 | 0 | \n| Crow | 1 | 0 | 0 | 0 | \n| Flying superhero | 1 | 1 | 0 | 0 | \n| swap-flight | 0 | 0 | 1 | 0 | \n| swap-speech | 0 | 0 | 0 | 1 | \n| keep-flight | 0 | 0 | 0 | 0 | \n| keep-speech | 0 | 0 | 0 | 0 | \n| he-is? | 0 | 0 | 0 | 0 | \n\n**Bits 5-8 - token type:**\n\n| Token | object? | flight act? | speech act? | question? | \n|---|---|---|---|---|\n| Rock | 1 | 0 | 0 | 0 | \n| Human | 1 | 0 | 0 | 0 | \n| Crow | 1 | 0 | 0 | 0 | \n| Flying superhero | 1 | 0 | 0 | 0 | \n| swap-flight | 0 | 1 | 0 | 0 | \n| swap-speech | 0 | 0 | 1 | 0 | \n| keep-flight | 0 | 1 | 0 | 0 | \n| keep-speech | 0 | 0 | 1 | 0 | \n| he-is? | 0 | 0 | 0 | 1 | \n\n```\nbit_labels = [\"can fly\", \"can speak\", \"swap flight\", \"swap speech\",\n        \"object?\", \"flight act?\", \"speech act?\", \"question?\"]\n\ntoken_to_bitvec = {\n    #                    fly spk swF swS  obj flA spA  q\n    \"Rock\":             [0,  0,  0,  0,   1,  0,  0,  0],\n    \"Human\":            [0,  1,  0,  0,   1,  0,  0,  0],\n    \"Crow\":             [1,  0,  0,  0,   1,  0,  0,  0],\n    \"Flying superhero\": [1,  1,  0,  0,   1,  0,  0,  0],\n    \"swap-flight\":      [0,  0,  1,  0,   0,  1,  0,  0],\n    \"swap-speech\":      [0,  0,  0,  1,   0,  0,  1,  0],\n    \"keep-flight\":      [0,  0,  0,  0,   0,  1,  0,  0],\n    \"keep-speech\":      [0,  0,  0,  0,   0,  0,  1,  0],\n    \"he-is?\":           [0,  0,  0,  0,   0,  0,  0,  1],\n}\n\ntoken_to_bitvec = {k: np.array(v) for k, v in token_to_bitvec.items()}\n\nfor tok, v in token_to_bitvec.items():\n    print(f\"{tok:18} {v[:4]}  {v[4:]}\")\nRock               [0 0 0 0]  [1 0 0 0]\nHuman              [0 1 0 0]  [1 0 0 0]\nCrow               [1 0 0 0]  [1 0 0 0]\nFlying superhero   [1 1 0 0]  [1 0 0 0]\nswap-flight        [0 0 1 0]  [0 1 0 0]\nswap-speech        [0 0 0 1]  [0 0 1 0]\nkeep-flight        [0 0 0 0]  [0 1 0 0]\nkeep-speech        [0 0 0 0]  [0 0 1 0]\nhe-is?             [0 0 0 0]  [0 0 0 1]\nsentence = [\"Crow\", \"keep-flight\", \"swap-speech\", \"he-is?\"]\n\nX = np.stack([token_to_bitvec[t] for t in sentence])\nprint(X.shape)\nX\n(4, 8)\n\narray([[1, 0, 0, 0, 1, 0, 0, 0],\n       [0, 0, 0, 0, 0, 1, 0, 0],\n       [0, 0, 0, 1, 0, 0, 1, 0],\n       [0, 0, 0, 0, 0, 0, 0, 1]])\n```\n\nSkip the equations and go to the hand-written Q, K, V weights to get the idea behind these equations first.\n\nN heads. Each head does:\n\n``` python\ndef softmax(z, axis=-1):\n    z = z - z.max(axis=axis, keepdims=True)\n    e = np.exp(z)\n    return e / e.sum(axis=axis, keepdims=True)\n\ndef head(X, Wq, Wk, Wv):\n    Q, K, V = X @ Wq, X @ Wk, X @ Wv\n    d_h = Wq.shape[1]\n    scores = Q @ K.T / np.sqrt(d_h)\n    A = softmax(scores)\n    return A @ V, A, Q, K, V\n```\n\nQ - Query. K - Key. V - Value (payload).\n\nThe Q, K, V are intermediate tensors, what the model has are the corresponding weights. \n\nThe weight matrices transform the input in three ways to get the (Q, K, V).\n\nX is the input. Its the full sentence in our example. A context length full of tokens as input. \n\nThe three weight matrices transform the input. The three matrices perform - three tasks. \n\nThe weight matrices are per-head. i.e each head can extract out different information from the tokens. \n\nWQ\n is weights for getting the queries for this head.\n\nWK\n are the weights that transform the input tokens to keys. (keys are like answers to the queries)\n\nWV\n is the payload, if the question and the answer for a pair of tokens get a high score, the W_V helps extract the payload or values from the tokens to pass along towards the FFN (toward the concatenation operation, then the FFN).\n\nEvery word asks a question; And also the question is not the same across heads. Each word in Each head can ask a different question; We have two heads in this hand-constructed example (Object Head, Action Verb-Object head)\n\nObject head - `he-is?` asks: are you an Object?\n\nKeys answer. Every token answers.\n\nObject head - \"Crow\" replies: I am an Object.\n\n`he-is?` in the object head asks - are you an object?\n\n```\nWq1 = np.zeros((8, 2)); Wq1[7] = [8, 0]\nWk1 = np.zeros((8, 2)); Wk1[4] = [1, 0]\nWv1 = np.zeros((8, 2)); Wv1[0] = [1, 0]; Wv1[1] = [0, 1]\n\nprint(\"Wq1\\n\", Wq1, \"\\n\\nWk1\\n\", Wk1, \"\\n\\nWv1\\n\", Wv1)\nWq1\n [[0. 0.]\n [0. 0.]\n [0. 0.]\n [0. 0.]\n [0. 0.]\n [0. 0.]\n [0. 0.]\n [8. 0.]] \n\nWk1\n [[0. 0.]\n [0. 0.]\n [0. 0.]\n [0. 0.]\n [1. 0.]\n [0. 0.]\n [0. 0.]\n [0. 0.]] \n\nWv1\n [[1. 0.]\n [0. 1.]\n [0. 0.]\n [0. 0.]\n [0. 0.]\n [0. 0.]\n [0. 0.]\n [0. 0.]]\nWq2 = np.zeros((8, 2)); Wq2[7] = [8, 8]\nWk2 = np.zeros((8, 2)); Wk2[5] = [1, 0]; Wk2[6] = [0, 1]\nWv2 = np.zeros((8, 2)); Wv2[2] = [1, 0]; Wv2[3] = [0, 1]\n\nprint(\"Wq2\\n\", Wq2, \"\\n\\nWk2\\n\", Wk2, \"\\n\\nWv2\\n\", Wv2)\nWq2\n [[0. 0.]\n [0. 0.]\n [0. 0.]\n [0. 0.]\n [0. 0.]\n [0. 0.]\n [0. 0.]\n [8. 8.]] \n\nWk2\n [[0. 0.]\n [0. 0.]\n [0. 0.]\n [0. 0.]\n [0. 0.]\n [1. 0.]\n [0. 1.]\n [0. 0.]] \n\nWv2\n [[0. 0.]\n [0. 0.]\n [1. 0.]\n [0. 1.]\n [0. 0.]\n [0. 0.]\n [0. 0.]\n [0. 0.]]\n```\n\nCrow, bits 1-8: `1 0 0 0 1 0 0 0`\n\nThe payload is Crow's object attributes.\n\n```\ncrow = token_to_bitvec[\"Crow\"]\n\nprint(\"x_Crow           \", crow)\nprint(\"x_Crow @ Wk1     \", crow @ Wk1, \"  <- key: I am an object\")\nprint(\"x_Crow @ Wv1     \", crow @ Wv1, \"  <- value: [can fly, can speak]\")\nx_Crow            [1 0 0 0 1 0 0 0]\nx_Crow @ Wk1      [1. 0.]   <- key: I am an object\nx_Crow @ Wv1      [1. 0.]   <- value: [can fly, can speak]\n```\n\nswap-flight, bits 1-8: `0 0 1 0 0 1 0 0`\n\nThe payload is the correction.\n\n```\nsf = token_to_bitvec[\"swap-flight\"]\n\nprint(\"x_swap-flight        \", sf)\nprint(\"x_swap-flight @ Wk2  \", sf @ Wk2, \"  <- key: I am a flight action\")\nprint(\"x_swap-flight @ Wv2  \", sf @ Wv2, \"  <- value: [swap flight, swap speech]\")\nx_swap-flight         [0 0 1 0 0 1 0 0]\nx_swap-flight @ Wk2   [1. 0.]   <- key: I am a flight action\nx_swap-flight @ Wv2   [1. 0.]   <- value: [swap flight, swap speech]\n```\n\n`he-is?` is the last token, so we read row 3 of the attention matrix.\n\n```\nout1, A1, Q1, K1, V1 = head(X, Wq1, Wk1, Wv1)\nout2, A2, Q2, K2, V2 = head(X, Wq2, Wk2, Wv2)\n\nq = sentence.index(\"he-is?\")\n\nprint(\"Object head - attention from he-is?\")\nfor tok, k, a in zip(sentence, K1, A1[q]):\n    print(f\"  {tok:14} key={k}  softmax={a:.2f}\")\nprint(\"  head 1 output:\", out1[q])\n\nprint(\"\\nAction head - attention from he-is?\")\nfor tok, k, a in zip(sentence, K2, A2[q]):\n    print(f\"  {tok:14} key={k}  softmax={a:.2f}\")\nprint(\"  head 2 output:\", out2[q])\nObject head - attention from he-is?\n  Crow           key=[1. 0.]  softmax=0.99\n  keep-flight    key=[0. 0.]  softmax=0.00\n  swap-speech    key=[0. 0.]  softmax=0.00\n  he-is?         key=[0. 0.]  softmax=0.00\n  head 1 output: [0.99 0.  ]\n\nAction head - attention from he-is?\n  Crow           key=[0. 0.]  softmax=0.00\n  keep-flight    key=[1. 0.]  softmax=0.50\n  swap-speech    key=[0. 1.]  softmax=0.50\n  he-is?         key=[0. 0.]  softmax=0.00\n  head 2 output: [0.  0.5]\n```\n\nTwo action words split the mass, so head 2 gives `[0, 0.5]`.\n\nWO\n scales by 2 -> `[0, 1]`.\n\n```\nWo = np.diag([1.0, 1.0, 2.0, 2.0])   # head 2 mass was split across 2 words\n\nconcat = np.concatenate([out1[q], out2[q]])\nfinal = concat @ Wo\n\nprint(\"concat        \", concat)\nprint(\"after W_O     \", final)\nprint()\nfor name, val in zip(bit_labels[:4], final):\n    print(f\"  {name:12} {val:.0f}\")\nconcat         [0.99 0.   0.   0.5 ]\nafter W_O      [0.99 0.   0.   1.  ]\n\n  can fly      1\n  can speak    0\n  swap flight  0\n  swap speech  1\n```\n\nHand-built weights pulled the object attributes and the\n\ncorrection attributes into 4 bits.\n\nCrow: flies, no speech. Correction: swap speech.\n\n-> Flying superhero. Same 4 bits the ANN wanted.\n\n```\nfly, speak, swap_fly, swap_speak = (int(v) for v in final.round())\n\nobj_in = (fly, 0, speak)                       # ANN bits: fly, wheels, speak\ncorrection = (swap_fly, 0, swap_speak)\n\nprint(\"object in :\", attributes_to_word[obj_in])\nprint(\"correction:\", correction)\nprint(\"object out:\", apply_correction(attributes_to_word[obj_in], correction))\nobject in : Crow\ncorrection: (0, 0, 1)\nobject out: Flying Superhero\n```\n\nThis notebook is part of the support files for the TechAarvam workshop\n\n**[Build Your Own Model](https://www.techaarvam.com/workshops/build-your-own-model)**.\n\n© TechAarvam. You are free to use, copy, modify, share and build on this\n\nmaterial, including for commercial purposes, **provided you credit TechAarvam** and link back to", "url": "https://wpnews.pro/news/attention-is-simpler-than-you-think-a-hand-crafted-superhero-transformer", "canonical_source": "https://dev.to/techaarvam/attention-is-simpler-than-you-think-a-hand-crafted-superhero-transformer-pid", "published_at": "2026-09-08 11:46:52+00:00", "updated_at": "2026-09-08 12:02:11.438178+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models"], "entities": ["TechAarvam"], "alternates": {"html": "https://wpnews.pro/news/attention-is-simpler-than-you-think-a-hand-crafted-superhero-transformer", "markdown": "https://wpnews.pro/news/attention-is-simpler-than-you-think-a-hand-crafted-superhero-transformer.md", "text": "https://wpnews.pro/news/attention-is-simpler-than-you-think-a-hand-crafted-superhero-transformer.txt", "jsonld": "https://wpnews.pro/news/attention-is-simpler-than-you-think-a-hand-crafted-superhero-transformer.jsonld"}}