{"slug": "gaussian-elimination-the-algorithm-hiding-inside-numpy-that-i-was-doing-by-hand", "title": "Gaussian Elimination: the algorithm hiding inside NumPy that I was doing by hand", "summary": "A backend engineer pivoting into AI engineering discovered that the Gaussian elimination algorithm they were practicing by hand is the same algorithm running inside NumPy's np.linalg.solve. The engineer realized that the linear algebra foundations they were studying are not just background knowledge but the actual substrate of machine learning systems, including backpropagation, attention mechanisms, and gradient descent.", "body_md": "There's a specific moment in studying math that hits different as an engineer: when you realize the \"academic exercise\" you're grinding through is literally running inside production software you've used for years.\n\nThat moment happened to me recently. I've been pivoting from backend engineering (TypeScript, NestJS, distributed systems) into AI Engineering, and I decided I wasn't going to fake my way through the math. No skipping the foundations. So I went back to Gilbert Strang's MIT 18.06 and started solving linear systems by hand. And then it clicked.\n\nI was working through a 3×3 system:\n\n```\nx  + 2y - z  = 3\n2x +  y + z  = 7\n3x -  y + 2z = 8\n```\n\nWhich becomes an augmented matrix:\n\n```\n[ 1  2 -1 | 3 ]\n[ 2  1  1 | 7 ]\n[ 3 -1  2 | 8 ]\n```\n\nThe goal: zero out everything below the diagonal. Pivot by pivot.\n\n**First pivot (column 1):**\n\n```\nL2 ← L2 - 2·L1  →  [ 0  -3   3 |  1 ]\nL3 ← L3 - 3·L1  →  [ 0  -7   5 | -1 ]\n```\n\n**Second pivot (column 2):**\n\n```\nL3 ← 3·L3 - 7·L2  →  [ 0  0  -6 | -10 ]\n```\n\nUpper triangular form:\n\n```\n[ 1  2  -1 |   3 ]\n[ 0 -3   3 |   1 ]\n[ 0  0  -6 | -10 ]\n```\n\nBack-substitution from bottom to top gives:\n\n```\nz = 5/3,  y = 4/3,  x = 2\n```\n\nStandard stuff. Nothing fancy. Or so I thought.\n\n`m`\n\nEvery elimination step computes a multiplier:\n\n```\nm = element_to_zero / pivot\n```\n\nSo when zeroing out `L2[0]`\n\nusing `L1`\n\nas pivot row:\n\n```\nm = 2/1 = 2  →  L2 ← L2 - 2·L1\n```\n\nFor `L3[0]`\n\n:\n\n```\nm = 3/1 = 3  →  L3 ← L3 - 3·L1\n```\n\nI was doing this mechanically, column by column, treating each operation as:\n\n```\nA[i][j] = A[i][j] - m * A[pivot][j]\n```\n\nThat's it. That's the whole thing. And that's when I looked at an algorithm and went quiet for a second.\n\n```\nfor pivot in range(n):\n    for row in range(pivot + 1, n):\n        m = A[row][pivot] / A[pivot][pivot]\n        A[row] = A[row] - m * A[pivot]\n```\n\nThe exact sequence I was doing by hand, pivot selection, multiplier computation, row update, is the algorithm. Not a simplification of it. Not a conceptual analogy. The actual algorithm.\n\nAnd when you call `np.linalg.solve(A, b)`\n\n, you're running a production-grade and optimized version of this. The math is the same. The performance engineering around it is what makes it fast.\n\nNumPy doesn't literally run Gaussian Elimination in the naive textbook form. What it actually computes under the hood is LU decomposition, a factorization of the matrix into two triangular pieces, where U is essentially what we produced with elimination, and L stores the multipliers m along the way.\n\nI haven't gone deep into LU yet. But understanding that the elimination I was doing by hand is the entry point to that decomposition changed how I see the abstraction. It's not magic. It's the same loop, formalized.\n\nI came in thinking I was filling a gap in my math background. I came out understanding something structural: the linear algebra I'm studying isn't background knowledge for ML, it *is* the substrate of ML.\n\nBackprop is the chain rule applied to matrix operations. Attention in transformers is matrix multiplication with a softmax. Embeddings live in vector spaces where distance and similarity are defined by inner products. The gradient descent step is a vector subtraction.\n\nWhen Gilbert Strang says \"the key ideas of linear algebra\" he's not being poetic. Those ideas are load-bearing walls in almost every ML system.\n\nI'm still early in this path, backend engineer moving into AI Engineering, currently building and studying simultaneously. But I'm increasingly convinced that the engineers who understand what's happening inside `np.linalg.solve`\n\nwill make better decisions than the ones who only know how to call it.\n\n*I'm documenting this pivot publicly. My RAG project is live at buscadegeloefogo.vercel.app, the Linear Algebra visualizer I built as a study tool is at github.com/FelipeAraujoBS/LA-Canva-Playground. More posts incoming as I go deeper.*", "url": "https://wpnews.pro/news/gaussian-elimination-the-algorithm-hiding-inside-numpy-that-i-was-doing-by-hand", "canonical_source": "https://dev.to/felipearaujobs/gaussian-elimination-the-algorithm-hiding-inside-numpy-that-i-was-doing-by-hand-1ahn", "published_at": "2026-06-18 13:06:42+00:00", "updated_at": "2026-06-18 13:21:31.803076+00:00", "lang": "en", "topics": ["machine-learning", "developer-tools", "ai-infrastructure"], "entities": ["NumPy", "Gilbert Strang", "MIT 18.06", "Felipe Araujo BS"], "alternates": {"html": "https://wpnews.pro/news/gaussian-elimination-the-algorithm-hiding-inside-numpy-that-i-was-doing-by-hand", "markdown": "https://wpnews.pro/news/gaussian-elimination-the-algorithm-hiding-inside-numpy-that-i-was-doing-by-hand.md", "text": "https://wpnews.pro/news/gaussian-elimination-the-algorithm-hiding-inside-numpy-that-i-was-doing-by-hand.txt", "jsonld": "https://wpnews.pro/news/gaussian-elimination-the-algorithm-hiding-inside-numpy-that-i-was-doing-by-hand.jsonld"}}