26 Important Concepts that you should learn in Linear Algebra for Machine Learning — PART II

A Medium blog post by Rajendran22 presents Part II of a two-part series on linear algebra for machine learning, covering advanced concepts such as inner products, orthogonality, projections, and the Gram-Schmidt process, with Python code examples. The post emphasizes the importance of mathematical understanding for machine learning and includes practical applications like dimensionality reduction and feature selection.

Linear Algebra, being one of the most important subjects, is essential for anyone who’s working with data and predictive algorithms. This is Part II of the two-part linear algebra series. You can read Part I of this series and continue with this blog. 26 Important Concepts that you should learn in Linear Algebra for Machine Learning — PART I https://rajendran22.medium.com/26-important-concepts-that-you-should-learn-in-linear-algebra-for-machine-learning-part-i-f87c8685da52 Ironically, most people ask how much math is enough for machine learning. While there is no universal quantitative answer available, it is evident that you need math to understand machine learning. At any point, if you feel you don’t understand any derivation or mathematical definition, you have to brush up on your mathematical concepts. If you are a complete beginner, this is a great place to start and get a bigger picture of how things work together. If you are already familiar with mathematics, this blog could help you to brush up and rewire your mathematical knowledge. The earlier part of this series covered the most fundamental concepts of linear algebra. This part intends to cover more advanced and interesting concepts, and it’s real-time applications in machine learning. Inner tells you how similar two vectors are and how much one vector “projects” onto another. python import numpy as npu = np.array 1, 2, 3 v = np.array 4, 5, 6 Inner productdot = np.dot u, v print "Dot product:", dot 1 4 + 2 5 + 3 6 = 32 Normsnorm u = np.linalg.norm u norm v = np.linalg.norm v Cosine similaritycos sim = dot / norm u norm v print "Cosine similarity:", cos sim Projection of u onto vproj = dot / norm v 2 vprint "Projection:", proj Two vectors are said to be orthogonal when their inner product is zero. In the context of machine learning: python import numpy as np Define two vectorsu = np.array 1, 0 v = np.array 0, 1 Check orthogonalitydot product = np.dot u, v print "Dot product:", dot product 0 → orthogonal Orthonormal vectorsu norm = u / np.linalg.norm u v norm = v / np.linalg.norm v print "Are they orthonormal?", np.isclose np.dot u norm, v norm , 0 A projection is an operation of mapping a vector onto a subspace. Intuitively, it is about dropping a high-dimensional vector onto a direction or plane. The projected vector is the closest point Euclidean distance to the original vector within the subspace. The projection is mainly used in linear regression, PCA dimensionality reduction, denoising, embedding, etc. In ML, “projection” generally refers to dimensionality reduction, where high-dimensional data is mapped to a lower-dimensional space to make it easier to visualize, analyze, or train models. python import numpy as npA = np.array 1, 1 , 1, 0 Q, R = np.linalg.qr A print "Orthonormal basis Q :" print Q The Gram-Schmidt process is used to convert a set of vectors into an orthonormal basis. There are two basic concepts: The Gram-Schmidt process accepts a set of linearly independent vectors and converts them into an orthonormal set. Key benefits of the Gram-Schmidt Process: 1. Numerical Stability 2. Dimensionality Reduction 3. Effective Matrix Decomposition 4. Feature Selection python import numpy as npdef gram schmidt V : """ Classical Gram-Schmidt process. Input: V is a matrix whose columns are the vectors v1, v2, ..., vk Output: Q is a matrix whose columns are the orthonormal basis u1, u2, ..., uk """ V = np.array V, dtype=float n, k = V.shape Q = np.zeros n, k for i in range k : Start with v i qi = V :, i Subtract projections onto previous q j for j in range i : proj = np.dot Q :, j , qi Q :, j qi = qi - proj Normalize Q :, i = qi / np.linalg.norm qi return QV = np.array 1, 1 , 1, 0 Q = gram schmidt V print "Orthonormal basis Classical GS :" print Q Eigenvalues and Eigenvectors in PCA python import numpy as np Example matrixA = np.array 4, 2 , 1, 3 Built-in functioneigenvalues, eigenvectors = np.linalg.eig A print "Eigenvalues:" print eigenvalues print "\nEigenvectors columns :" print eigenvectors verify the eigen equationfor i in range len eigenvalues : v = eigenvectors :, i λ = eigenvalues i print "\nCheck for eigenvalue:", λ print "A @ v:", A @ v print "λ v:", λ v python import numpy as npA = np.array 4, 1 , 2, 3 Step 1: eigenvalues & eigenvectorseigenvalues, eigenvectors = np.linalg.eig A Step 2: Construct D diagonal matrix D = np.diag eigenvalues Step 3: Matrix P eigenvectors P = eigenvectors Step 4: Find P inverseP inv = np.linalg.inv P Step 5: Reconstruct A should be equal to original A A reconstructed = P @ D @ P invprint "Eigenvalues:\n", eigenvalues print "Eigenvectors P :\n", P print "Diagonal matrix D:\n", D print "P inverse:\n", P inv print "Reconstructed A:\n", A reconstructed python import numpy as np Symmetric matrixA = np.array 2, 1, 0 , 1, 2, 1 , 0, 1, 2 Verify symmetryprint "Is symmetric:", np.allclose A, A.T Eigen decompositioneigvals, eigvecs = np.linalg.eigh A eigh is optimized for symmetric/hermitianprint "Eigenvalues:", eigvals print "Eigenvectors columns :\n", eigvecs Verify orthogonality of eigenvectorsprint "Q^T Q:\n", eigvecs.T @ eigvecs Should be identity python import numpy as npimport matplotlib.pyplot as plt Example: random 5x4 matrixX = np.random.randn 5, 4 SVDU, S, VT = np.linalg.svd X, full matrices=False print "U:\n", U print "Singular values:", S print "V^T:\n", VT Rankrank = np.sum S 1e-10 print "Rank of X:", rank Low-rank approximation keep top 2 singular values k = 2U k = U :, :k S k = np.diag S :k VT k = VT :k, : X approx = U k @ S k @ VT kprint "Low-rank approx:\n", X approx Norms are directly built from inner products: Why it’s important: This pairs naturally with inner products since the two are mathematically linked. Cosine similarity measures the similarity between two non-zero vectors by calculating the cosine of the angle between them. Similarity measure calculates the distance between data objects based on their feature dimensions in a dataset. A smaller distance indicates a higher similarity, while a larger distance indicates a lower similarity. Cosine similarity is a metric, helpful in determining how similar the data objects are, irrespective of their size. This is equivalent to the matrix having all non-negative eigenvalues Understanding PSD matrices gives insight into: “Machine learning is nothing without mathematics” — I have heard a lot of them saying this. However, I truly understood it only when I deep dived into the machine learning algorithms. A solid understanding of linear algebra concepts helps to understand how machine learning works in a better and effective way. With that, this blog comes to an end. This is only the beginning I’m also planning to write detailed articles on probability, statistics, and calculus for machine learning , so stay tuned for more posts that will help you strengthen your mathematical foundation for ML. If this series helped you, consider following me to get updates — there’s plenty more interesting stuff on the way. Note: Most of the content is represented as images due to the lack of LaTeX compatibility in Medium. 26 Important Concepts that you should learn in Linear Algebra for Machine Learning — PART II https://pub.towardsai.net/26-important-concepts-that-you-should-learn-in-linear-algebra-for-machine-learning-part-ii-8110461624e3 was originally published in Towards AI https://pub.towardsai.net on Medium, where people are continuing the conversation by highlighting and responding to this story.