{"slug": "puzzle-solution-revealed-transformer-need-for-position-embedding", "title": "Puzzle Solution Revealed - Transformer: Need for Position Embedding", "summary": "A developer has published a solution to a puzzle from a workshop on building a minimal hand-constructed transformer, demonstrating why position embeddings and residual connections are necessary. The extended model adds a 'disobeys' token that modifies only the immediately following word, forcing word order to matter, and introduces a 22-bit residual stream carrying position one-hot encodings plus each layer's findings. The author notes that while real implementations learn vector and position embeddings via gradient descent, the hand-constructed example clarifies why those blocks and connections exist.", "body_md": "▶ [Watch the 7-minute walkthrough](https://youtu.be/Hav2zlqxzUI)\n\n``` python\nimport torch\n\ntorch.set_printoptions(precision=2, sci_mode=False, linewidth=160)\n```\n\nIn this notebook we are extending the previous notebook, [attention_ann.ipynb](https://colab.research.google.com/github/techaarvam/byom_workshop/blob/main/attention_ann.ipynb), which introduced (plausibly) the world's tiniest hand-constructed transformer model. Please read the previous notebook for the context. This is the solution to the puzzle that was introduced as part of the previous notebook.\n\nWe add one word 'disobeys' which modifies the action-attribute words (swap-speech, keep-speech, swap-flight, keep-flight). In the previous notebook the order of the words did not matter. In the current notebook, the order does matter. The word disobeys modifies only the word right after it, so word order matters.\n\nThe solution adds an extra layer, and a residual connection that carries the input unmodified to the next block, so each next layer gets both the unmodified input and the modified input.\n\nThe residual stream is made 22 bits, where it carries the original inputs and each layer's/head's findings. The most interesting addition is the position information. Each token carries the position where it appears.\n\nThis example is constructed to illustrate the ideas. In the real implementation, we do not hand-construct in this manner. Vector embedding and position embedding are often also learned using the training loop and gradient descent. But the hand-construction allows us to see why those blocks and connections exist and how they are helpful to have.\n\n| idx | slot | \n|---|---|\n| 0 | `fly` | \n| 1 | `speak` | \n| 2 | `swap_fly` | \n| 3 | `swap_speak` | \n| 4 | `object` | \n| 5 | `action_fly` | \n| 6 | `action_speak` | \n| 7 | `question` | \n| 8 | `disobey` | \n| 9 | `pos0` | \n| 10 | `pos1` | \n| 11 | `pos2` | \n| 12 | `pos3` | \n| 13 | `pos4` | \n| 14 | `pos5` | \n| 15 | `previous_word_is_disobey` | \n| 16 | `object_attribute_fly` | \n| 17 | `object_attribute_speak` | \n| 18 | `is_swap_attr_fly` | \n| 19 | `is_attr_fly_disobeyed` | \n| 20 | `is_swap_attr_speak` | \n| 21 | `is_attr_speak_disobeyed` | \n\n```\n# The new list of token bits, with the additional word 'disobeys'\nidx = {\"fly\": 0, \"speak\": 1, \"swap_fly\": 2, \"swap_speak\": 3,\n       \"object\": 4, \"action_fly\": 5, \"action_speak\": 6, \"question\": 7, \"disobey\": 8}\n\n# constants used for bit-slicing and locating the portion of\n# the residual we need\n\n# L is the maximum sentence length, i.e. the number of position slots.\n# It is used to name the position slots below, and to build the\n# \"Disobey Position Finder\" head (Wq_P, Wk_P) in layer 1.\nposition_start, L = 9, 6             # position one-hot occupies 9 .. 14\nprevious_word_is_disobey = 15\nobject_attribute_fly, object_attribute_speak = 16, 17\nis_swap_attr_fly, is_attr_fly_disobeyed = 18, 19\nis_swap_attr_speak, is_attr_speak_disobeyed = 20, 21\nnum_bits = 22\n\nslot_names = {v: k for k, v in idx.items()}\nslot_names.update({position_start + p: f\"pos{p}\" for p in range(L)})\nslot_names.update({previous_word_is_disobey: \"previous_word_is_disobey\",\n                   object_attribute_fly: \"object_attribute_fly\",\n                   object_attribute_speak: \"object_attribute_speak\",\n                   is_swap_attr_fly: \"is_swap_attr_fly\",\n                   is_attr_fly_disobeyed: \"is_attr_fly_disobeyed\",\n                   is_swap_attr_speak: \"is_swap_attr_speak\",\n                   is_attr_speak_disobeyed: \"is_attr_speak_disobeyed\"})\n\nfor i in range(num_bits):\n    print(f\"{i:2} {slot_names[i]}\")\n0 fly\n 1 speak\n 2 swap_fly\n 3 swap_speak\n 4 object\n 5 action_fly\n 6 action_speak\n 7 question\n 8 disobey\n 9 pos0\n10 pos1\n11 pos2\n12 pos3\n13 pos4\n14 pos5\n15 previous_word_is_disobey\n16 object_attribute_fly\n17 object_attribute_speak\n18 is_swap_attr_fly\n19 is_attr_fly_disobeyed\n20 is_swap_attr_speak\n21 is_attr_speak_disobeyed\ntoken_to_vector = {\n    #                   fly spk swF swS  obj actF actS  q  dis\n    \"Rock\":             [0,  0,  0,  0,   1,  0,  0,  0,  0],\n    \"Human\":            [0,  1,  0,  0,   1,  0,  0,  0,  0],\n    \"Crow\":             [1,  0,  0,  0,   1,  0,  0,  0,  0],\n    \"Flying superhero\": [1,  1,  0,  0,   1,  0,  0,  0,  0],\n    \"swap-flight\":      [0,  0,  1,  0,   0,  1,  0,  0,  0],\n    \"swap-speech\":      [0,  0,  0,  1,   0,  0,  1,  0,  0],\n    \"keep-flight\":      [0,  0,  0,  0,   0,  1,  0,  0,  0],\n    \"keep-speech\":      [0,  0,  0,  0,   0,  0,  1,  0,  0],\n    \"disobeys\":         [0,  0,  0,  0,   0,  0,  0,  0,  1],\n    \"he-is?\":           [0,  0,  0,  0,   0,  0,  0,  1,  0],\n}\n\nfor tok, bits in token_to_vector.items():\n    print(f\"{tok:18} {torch.tensor(bits)}\")\nRock               tensor([0, 0, 0, 0, 1, 0, 0, 0, 0])\nHuman              tensor([0, 1, 0, 0, 1, 0, 0, 0, 0])\nCrow               tensor([1, 0, 0, 0, 1, 0, 0, 0, 0])\nFlying superhero   tensor([1, 1, 0, 0, 1, 0, 0, 0, 0])\nswap-flight        tensor([0, 0, 1, 0, 0, 1, 0, 0, 0])\nswap-speech        tensor([0, 0, 0, 1, 0, 0, 1, 0, 0])\nkeep-flight        tensor([0, 0, 0, 0, 0, 1, 0, 0, 0])\nkeep-speech        tensor([0, 0, 0, 0, 0, 0, 1, 0, 0])\ndisobeys           tensor([0, 0, 0, 0, 0, 0, 0, 0, 1])\nhe-is?             tensor([0, 0, 0, 0, 0, 0, 0, 1, 0])\npython\ndef embed(sentence):\n    X = torch.zeros(len(sentence), num_bits)\n    for current_token_position, current_token in enumerate(sentence):\n        X[current_token_position, :9] = torch.tensor(token_to_vector[current_token], dtype=torch.float32)\n        X[current_token_position, position_start + current_token_position] = 1\n    return X\n\nsentence = [\"Crow\", \"disobeys\", \"keep-flight\", \"swap-speech\", \"he-is?\"]\nX = embed(sentence)\n\nfor tok, row in zip(sentence, X):\n    print(f\"{tok:14} {row[:9]}  {row[position_start:position_start + L]}  {row[previous_word_is_disobey:]}\")\nCrow           tensor([1., 0., 0., 0., 1., 0., 0., 0., 0.])  tensor([1., 0., 0., 0., 0., 0.])  tensor([0., 0., 0., 0., 0., 0., 0.])\ndisobeys       tensor([0., 0., 0., 0., 0., 0., 0., 0., 1.])  tensor([0., 1., 0., 0., 0., 0.])  tensor([0., 0., 0., 0., 0., 0., 0.])\nkeep-flight    tensor([0., 0., 0., 0., 0., 1., 0., 0., 0.])  tensor([0., 0., 1., 0., 0., 0.])  tensor([0., 0., 0., 0., 0., 0., 0.])\nswap-speech    tensor([0., 0., 0., 1., 0., 0., 1., 0., 0.])  tensor([0., 0., 0., 1., 0., 0.])  tensor([0., 0., 0., 0., 0., 0., 0.])\nhe-is?         tensor([0., 0., 0., 0., 0., 0., 0., 1., 0.])  tensor([0., 0., 0., 0., 1., 0.])  tensor([0., 0., 0., 0., 0., 0., 0.])\npython\ndef softmax(z, dim=-1):\n    z = z - z.max(dim=dim, keepdim=True).values\n    e = torch.exp(z)\n    return e / e.sum(dim=dim, keepdim=True)\n\ndef head(X, Wq, Wk, Wv):\n    Q, K, V = X @ Wq, X @ Wk, X @ Wv\n    A = softmax(Q @ K.T / Wq.shape[1] ** 0.5)\n    return A @ V, A\nWq_O = torch.zeros(num_bits, 2); Wq_O[idx[\"question\"]] = torch.tensor([8.0, 0.0])\nWk_O = torch.zeros(num_bits, 2); Wk_O[idx[\"object\"]]   = torch.tensor([1.0, 0.0])\nWv_O = torch.zeros(num_bits, 2); Wv_O[idx[\"fly\"]] = torch.tensor([1.0, 0.0]); Wv_O[idx[\"speak\"]] = torch.tensor([0.0, 1.0])\nWo_O = torch.zeros(2, num_bits); Wo_O[0, object_attribute_fly] = 1; Wo_O[1, object_attribute_speak] = 1\n\nprint(\"Wq_O nonzero rows:\", (Wq_O != 0).any(1).nonzero().flatten())\nprint(\"Wk_O nonzero rows:\", (Wk_O != 0).any(1).nonzero().flatten())\nprint(\"Wv_O nonzero rows:\", (Wv_O != 0).any(1).nonzero().flatten())\nprint(\"Wo_O nonzero cols:\", (Wo_O != 0).any(0).nonzero().flatten())\nWq_O nonzero rows: tensor([7])\nWk_O nonzero rows: tensor([4])\nWv_O nonzero rows: tensor([0, 1])\nWo_O nonzero cols: tensor([16, 17])\nS = 24.0\n\nWq_P = torch.zeros(num_bits, L)\nfor p in range(1, L):\n    Wq_P[position_start + p, p - 1] = S\n\nWk_P = torch.zeros(num_bits, L)\nfor p in range(L):\n    Wk_P[position_start + p, p] = 1\n\nWv_P = torch.zeros(num_bits, 1); Wv_P[idx[\"disobey\"], 0] = 1\nWo_P = torch.zeros(1, num_bits); Wo_P[0, previous_word_is_disobey] = 1\n\nM = Wq_P @ Wk_P.T\nprint(\"M[9:15, 9:15] =\")\nprint(M[position_start:position_start + L, position_start:position_start + L])\nprint(\"symmetric:\", torch.allclose(M, M.T))\nM[9:15, 9:15] =\ntensor([[ 0.,  0.,  0.,  0.,  0.,  0.],\n        [24.,  0.,  0.,  0.,  0.,  0.],\n        [ 0., 24.,  0.,  0.,  0.,  0.],\n        [ 0.,  0., 24.,  0.,  0.,  0.],\n        [ 0.,  0.,  0., 24.,  0.,  0.],\n        [ 0.,  0.,  0.,  0., 24.,  0.]])\nsymmetric: False\npython\ndef layer1(X):\n    oO, AO = head(X, Wq_O, Wk_O, Wv_O)\n    oP, AP = head(X, Wq_P, Wk_P, Wv_P)\n    X1 = X + oO @ Wo_O + oP @ Wo_P\n    return X1, AO, AP\n\nX1, AO, AP = layer1(X)\n\nprint(\"A_P\")\nprint(AP)\nprint()\nfor tok, row in zip(sentence, X1):\n    print(f\"{tok:14} previous_word_is_disobey={row[previous_word_is_disobey]:.3f}   \"\n          f\"object_attribute_fly={row[object_attribute_fly]:.3f}  \"\n          f\"object_attribute_speak={row[object_attribute_speak]:.3f}\")\nA_P\ntensor([[0.20, 0.20, 0.20, 0.20, 0.20],\n        [1.00, 0.00, 0.00, 0.00, 0.00],\n        [0.00, 1.00, 0.00, 0.00, 0.00],\n        [0.00, 0.00, 1.00, 0.00, 0.00],\n        [0.00, 0.00, 0.00, 1.00, 0.00]])\n\nCrow           previous_word_is_disobey=0.200   object_attribute_fly=0.200  object_attribute_speak=0.000\ndisobeys       previous_word_is_disobey=0.000   object_attribute_fly=0.200  object_attribute_speak=0.000\nkeep-flight    previous_word_is_disobey=1.000   object_attribute_fly=0.200  object_attribute_speak=0.000\nswap-speech    previous_word_is_disobey=0.000   object_attribute_fly=0.200  object_attribute_speak=0.000\nhe-is?         previous_word_is_disobey=0.000   object_attribute_fly=0.986  object_attribute_speak=0.000\n```\n\nLayer 2 has two heads: get flight attribute and get speech attribute.\n\nUsually layers have a similar topology, so two heads are used in both layers.\n\nCould this work be done with a single head, like the action head in the previous notebook? Not with this residual layout. In the previous notebook, one head attended to both action words, and that worked because each action word's own swap bit (`swap_fly` or `swap_speak`) says which attribute it swaps. Here, the disobey information sits in one shared slot, `previous_word_is_disobey`, on both action words. A single head attending to both action words would add the two disobey signals into the same number, and they could no longer be told apart. For example, `Crow disobeys keep-flight swap-speech he-is?` and `Crow keep-flight disobeys swap-speech he-is?` would give exactly the same head output, but the answers are Human and Crow. So we use one head for the flight word and one head for the speech word.\n\n```\nWq_fly_head = torch.zeros(num_bits, 1); Wq_fly_head[idx[\"question\"], 0] = 8\nWk_fly_head = torch.zeros(num_bits, 1); Wk_fly_head[idx[\"action_fly\"], 0] = 1\nWv_fly_head = torch.zeros(num_bits, 2); Wv_fly_head[idx[\"swap_fly\"]] = torch.tensor([1.0, 0.0]); Wv_fly_head[previous_word_is_disobey] = torch.tensor([0.0, 1.0])\n\nWq_speak_head = torch.zeros(num_bits, 1); Wq_speak_head[idx[\"question\"], 0] = 8\nWk_speak_head = torch.zeros(num_bits, 1); Wk_speak_head[idx[\"action_speak\"], 0] = 1\nWv_speak_head = torch.zeros(num_bits, 2); Wv_speak_head[idx[\"swap_speak\"]] = torch.tensor([1.0, 0.0]); Wv_speak_head[previous_word_is_disobey] = torch.tensor([0.0, 1.0])\n\nWo_2 = torch.zeros(4, num_bits)\nWo_2[0, is_swap_attr_fly]        = 1\nWo_2[1, is_attr_fly_disobeyed]   = 1\nWo_2[2, is_swap_attr_speak]      = 1\nWo_2[3, is_attr_speak_disobeyed] = 1\n\nprint(\"Wv_fly_head nonzero rows:  \", (Wv_fly_head != 0).any(1).nonzero().flatten())\nprint(\"Wv_speak_head nonzero rows:\", (Wv_speak_head != 0).any(1).nonzero().flatten())\nprint(\"Wo_2 nonzero cols:         \", (Wo_2 != 0).any(0).nonzero().flatten())\nWv_fly_head nonzero rows:   tensor([ 2, 15])\nWv_speak_head nonzero rows: tensor([ 3, 15])\nWo_2 nonzero cols:          tensor([18, 19, 20, 21])\npython\ndef layer2_attn(X1):\n    oF, AF = head(X1, Wq_fly_head, Wk_fly_head, Wv_fly_head)\n    oS, AS = head(X1, Wq_speak_head, Wk_speak_head, Wv_speak_head)\n    X2 = X1 + torch.cat([oF, oS], dim=1) @ Wo_2\n    return X2, AF, AS\n\nX2, AF, AS = layer2_attn(X1)\nq = sentence.index(\"he-is?\")\n\nprint(\"A_F[q]\", AF[q])\nprint(\"A_S[q]\", AS[q])\nprint()\nprint(\"is_swap_attr_fly       \", X2[q, is_swap_attr_fly])\nprint(\"is_attr_fly_disobeyed  \", X2[q, is_attr_fly_disobeyed])\nprint(\"is_swap_attr_speak     \", X2[q, is_swap_attr_speak])\nprint(\"is_attr_speak_disobeyed\", X2[q, is_attr_speak_disobeyed])\nA_F[q] tensor([0.00, 0.00, 1.00, 0.00, 0.00])\nA_S[q] tensor([0.00, 0.00, 0.00, 1.00, 0.00])\n\nis_swap_attr_fly        tensor(0.)\nis_attr_fly_disobeyed   tensor(1.00)\nis_swap_attr_speak      tensor(1.00)\nis_attr_speak_disobeyed tensor(0.00)\n```\n\nThe FFN is shown for completeness, also as a hand-constructed implementation. But for understanding the ideas of attention, position embedding, residuals and the need for layers, this part can be skipped.\n\nSummary: bias values are used carefully to allow distinguishing 0, 1, 2, 3. This provides different ReLU activation levels corresponding to the number of flips. This FFN is a parity finder, while the FFN used in the previous notebook without the disobeys word was an XOR gate. Repeating the note that, in the real implementation, all the weights and biases are learned using gradient descent and the backpropagation algorithm.\n\n```\nW1 = torch.zeros(num_bits, 8)\nb1 = torch.zeros(8)\n\nfor r in (object_attribute_fly, is_swap_attr_fly, is_attr_fly_disobeyed):\n    W1[r, 0:4] = 1\nfor r in (object_attribute_speak, is_swap_attr_speak, is_attr_speak_disobeyed):\n    W1[r, 4:8] = 1\n\nb1[0:4] = torch.tensor([0.0, -1.0, -2.0, -3.0])\nb1[4:8] = torch.tensor([0.0, -1.0, -2.0, -3.0])\n\nW2 = torch.zeros(8, 2)\nW2[0:4, 0] = torch.tensor([1.0, -2.0, 2.0, -2.0])\nW2[4:8, 1] = torch.tensor([1.0, -2.0, 2.0, -2.0])\n\nprint(\"W1.T\\n\", W1.T)\nprint(\"\\nb1\", b1)\nprint(\"\\nW2.T\\n\", W2.T)\nprint(\"\\npi:\", [float(torch.relu(torch.tensor([s, s - 1.0, s - 2.0, s - 3.0])) @ W2[0:4, 0]) for s in range(4)])\nW1.T\n tensor([[0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 1., 1., 0., 0.],\n        [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 1., 1., 0., 0.],\n        [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 1., 1., 0., 0.],\n        [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 1., 1., 0., 0.],\n        [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 1., 1.],\n        [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 1., 1.],\n        [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 1., 1.],\n        [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 1., 1.]])\n\nb1 tensor([ 0., -1., -2., -3.,  0., -1., -2., -3.])\n\nW2.T\n tensor([[ 1., -2.,  2., -2.,  0.,  0.,  0.,  0.],\n        [ 0.,  0.,  0.,  0.,  1., -2.,  2., -2.]])\n\npi: [0.0, 1.0, 0.0, 1.0]\npython\ndef forward(sentence):\n    X = embed(sentence)\n    X1, AO, AP = layer1(X)\n    X2, AF, AS = layer2_attn(X1)\n    H = torch.relu(X2 @ W1 + b1)\n    Y = H @ W2\n    return dict(X=X, X1=X1, X2=X2, Y=Y, AO=AO, AP=AP, AF=AF, AS=AS)\n\nword_to_attributes = {\n    \"Rock\":              (0, 0, 0),\n    \"Human\":             (0, 0, 1),\n    \"Car\":               (0, 1, 0),\n    \"Talking Tow Truck\": (0, 1, 1),\n    \"Crow\":              (1, 0, 0),\n    \"Flying Superhero\":  (1, 0, 1),\n    \"Plane\":             (1, 1, 0),\n    \"Talking Planes\":    (1, 1, 1),\n}\nattributes_to_word = {b: w for w, b in word_to_attributes.items()}\n\ndef readout(sentence):\n    r = forward(sentence)\n    q = sentence.index(\"he-is?\")\n    fly, speak = (int(v.round()) for v in r[\"Y\"][q])\n    return attributes_to_word[(fly, 0, speak)], r[\"Y\"][q]\nsentences = [\n    [\"Human\", \"disobeys\", \"keep-flight\", \"disobeys\", \"swap-speech\", \"he-is?\"],\n    [\"Crow\", \"disobeys\", \"keep-flight\", \"swap-speech\", \"he-is?\"],\n    [\"Crow\", \"keep-flight\", \"disobeys\", \"swap-speech\", \"he-is?\"],\n    [\"Crow\", \"disobeys\", \"keep-flight\", \"disobeys\", \"swap-speech\", \"he-is?\"],\n]\n\nfor s in sentences:\n    word, y = readout(s)\n    print(f\"{' '.join(s):58} {y}  ->  {word}\")\nHuman disobeys keep-flight disobeys swap-speech he-is?     tensor([1.00, 0.98])  ->  Flying Superhero\nCrow disobeys keep-flight swap-speech he-is?               tensor([0.02, 1.00])  ->  Human\nCrow keep-flight disobeys swap-speech he-is?               tensor([0.99, 0.00])  ->  Crow\nCrow disobeys keep-flight disobeys swap-speech he-is?      tensor([0.02, 0.00])  ->  Rock\na = [\"Crow\", \"disobeys\", \"keep-flight\", \"swap-speech\", \"he-is?\"]\nb = [\"Crow\", \"keep-flight\", \"disobeys\", \"swap-speech\", \"he-is?\"]\n\nprint(sorted(a) == sorted(b))\nprint(readout(a)[0])\nprint(readout(b)[0])\nTrue\nHuman\nCrow\nfor s in sentences:\n    r = forward(s)\n    print(\" \".join(s))\n    print(\"  A_P\")\n    for tok, row in zip(s, r[\"AP\"]):\n        print(f\"    {tok:14} {row}\")\n    print(\"  previous_word_is_disobey\", r[\"X1\"][:, previous_word_is_disobey])\n    q = s.index(\"he-is?\")\n    print(\"  gathered  \", r[\"X2\"][q, [is_swap_attr_fly, is_attr_fly_disobeyed,\n                                      is_swap_attr_speak, is_attr_speak_disobeyed]])\n    print(\"  Y         \", r[\"Y\"][q])\n    print()\nHuman disobeys keep-flight disobeys swap-speech he-is?\n  A_P\n    Human          tensor([0.17, 0.17, 0.17, 0.17, 0.17, 0.17])\n    disobeys       tensor([1.00, 0.00, 0.00, 0.00, 0.00, 0.00])\n    keep-flight    tensor([0.00, 1.00, 0.00, 0.00, 0.00, 0.00])\n    disobeys       tensor([0.00, 0.00, 1.00, 0.00, 0.00, 0.00])\n    swap-speech    tensor([0.00, 0.00, 0.00, 1.00, 0.00, 0.00])\n    he-is?         tensor([0.00, 0.00, 0.00, 0.00, 1.00, 0.00])\n  previous_word_is_disobey tensor([0.33, 0.00, 1.00, 0.00, 1.00, 0.00])\n  gathered   tensor([0.00, 1.00, 1.00, 1.00])\n  Y          tensor([1.00, 0.98])\n\nCrow disobeys keep-flight swap-speech he-is?\n  A_P\n    Crow           tensor([0.20, 0.20, 0.20, 0.20, 0.20])\n    disobeys       tensor([1.00, 0.00, 0.00, 0.00, 0.00])\n    keep-flight    tensor([0.00, 1.00, 0.00, 0.00, 0.00])\n    swap-speech    tensor([0.00, 0.00, 1.00, 0.00, 0.00])\n    he-is?         tensor([0.00, 0.00, 0.00, 1.00, 0.00])\n  previous_word_is_disobey tensor([0.20, 0.00, 1.00, 0.00, 0.00])\n  gathered   tensor([0.00, 1.00, 1.00, 0.00])\n  Y          tensor([0.02, 1.00])\n\nCrow keep-flight disobeys swap-speech he-is?\n  A_P\n    Crow           tensor([0.20, 0.20, 0.20, 0.20, 0.20])\n    keep-flight    tensor([1.00, 0.00, 0.00, 0.00, 0.00])\n    disobeys       tensor([0.00, 1.00, 0.00, 0.00, 0.00])\n    swap-speech    tensor([0.00, 0.00, 1.00, 0.00, 0.00])\n    he-is?         tensor([0.00, 0.00, 0.00, 1.00, 0.00])\n  previous_word_is_disobey tensor([0.20, 0.00, 0.00, 1.00, 0.00])\n  gathered   tensor([0.00, 0.00, 1.00, 1.00])\n  Y          tensor([0.99, 0.00])\n\nCrow disobeys keep-flight disobeys swap-speech he-is?\n  A_P\n    Crow           tensor([0.17, 0.17, 0.17, 0.17, 0.17, 0.17])\n    disobeys       tensor([1.00, 0.00, 0.00, 0.00, 0.00, 0.00])\n    keep-flight    tensor([0.00, 1.00, 0.00, 0.00, 0.00, 0.00])\n    disobeys       tensor([0.00, 0.00, 1.00, 0.00, 0.00, 0.00])\n    swap-speech    tensor([0.00, 0.00, 0.00, 1.00, 0.00, 0.00])\n    he-is?         tensor([0.00, 0.00, 0.00, 0.00, 1.00, 0.00])\n  previous_word_is_disobey tensor([0.33, 0.00, 1.00, 0.00, 1.00, 0.00])\n  gathered   tensor([0.00, 1.00, 1.00, 1.00])\n  Y          tensor([0.02, 0.00])\n```\n\nThis notebook is part of the support files for the TechAarvam workshop\n\n**[Build Your Own Model](https://www.techaarvam.com/workshops/build-your-own-model)**.\n\n© TechAarvam. You are free to use, copy, modify, share and build on this\n\nmaterial, including for commercial purposes, **provided you credit TechAarvam** and link back to", "url": "https://wpnews.pro/news/puzzle-solution-revealed-transformer-need-for-position-embedding", "canonical_source": "https://dev.to/techaarvam/puzzle-solution-revealed-transformer-need-for-position-embedding-3k50", "published_at": "2026-09-15 13:05:50+00:00", "updated_at": "2026-09-15 13:14:01.876319+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "neural-networks", "ai-research"], "entities": ["Transformer", "PyTorch", "byom_workshop"], "alternates": {"html": "https://wpnews.pro/news/puzzle-solution-revealed-transformer-need-for-position-embedding", "markdown": "https://wpnews.pro/news/puzzle-solution-revealed-transformer-need-for-position-embedding.md", "text": "https://wpnews.pro/news/puzzle-solution-revealed-transformer-need-for-position-embedding.txt", "jsonld": "https://wpnews.pro/news/puzzle-solution-revealed-transformer-need-for-position-embedding.jsonld"}}