{"slug": "26-important-concepts-that-you-should-learn-in-linear-algebra-for-machine-part", "title": "26 Important Concepts that you should learn in Linear Algebra for Machine Learning — PART II", "summary": "A Medium blog post by Rajendran22 presents Part II of a two-part series on linear algebra for machine learning, covering advanced concepts such as inner products, orthogonality, projections, and the Gram-Schmidt process, with Python code examples. The post emphasizes the importance of mathematical understanding for machine learning and includes practical applications like dimensionality reduction and feature selection.", "body_md": "Linear Algebra, being one of the most important subjects, is essential for anyone who’s working with data and predictive algorithms. This is Part II of the two-part linear algebra series.\n\nYou can read Part I of this series and continue with this blog.\n\n[26 Important Concepts that you should learn in Linear Algebra for Machine Learning — PART I](https://rajendran22.medium.com/26-important-concepts-that-you-should-learn-in-linear-algebra-for-machine-learning-part-i-f87c8685da52)\n\nIronically, most people ask how much math is enough for machine learning. While there is no universal quantitative answer available, it is evident that you need math to understand machine learning. At any point, if you feel you don’t understand any derivation or mathematical definition, you have to brush up on your mathematical concepts.\n\nIf you are a complete beginner, this is a great place to start and get a bigger picture of how things work together. If you are already familiar with mathematics, this blog could help you to brush up and rewire your mathematical knowledge. The earlier part of this series covered the most fundamental concepts of linear algebra. This part intends to cover more advanced and interesting concepts, and it’s real-time applications in machine learning.\n\nInner tells you **how similar two vectors are** and how much one vector “projects” onto another.\n\n``` python\nimport numpy as npu = np.array([1, 2, 3])v = np.array([4, 5, 6])# Inner productdot = np.dot(u, v)print(\"Dot product:\", dot)  # 1*4 + 2*5 + 3*6 = 32# Normsnorm_u = np.linalg.norm(u)norm_v = np.linalg.norm(v)# Cosine similaritycos_sim = dot / (norm_u * norm_v)print(\"Cosine similarity:\", cos_sim)# Projection of u onto vproj = (dot / (norm_v**2)) * vprint(\"Projection:\", proj)\n```\n\nTwo vectors are said to be orthogonal when their inner product is zero.\n\n**In the context of machine learning:**\n\n``` python\nimport numpy as np# Define two vectorsu = np.array([1, 0])v = np.array([0, 1])# Check orthogonalitydot_product = np.dot(u, v)print(\"Dot product:\", dot_product)  # 0 → orthogonal# Orthonormal vectorsu_norm = u / np.linalg.norm(u)v_norm = v / np.linalg.norm(v)print(\"Are they orthonormal?\", np.isclose(np.dot(u_norm, v_norm), 0))\n```\n\nA projection is an operation of mapping a vector onto a subspace. Intuitively, it is about dropping a high-dimensional vector onto a direction or plane. The projected vector is the closest point (Euclidean distance) to the original vector within the subspace.\n\nThe projection is mainly used in linear regression, PCA dimensionality reduction, denoising, embedding, etc.\n\nIn ML, “projection” generally refers to dimensionality reduction, where high-dimensional data is mapped to a lower-dimensional space to make it easier to visualize, analyze, or train models.\n\n``` python\nimport numpy as npA = np.array([    [1, 1],    [1, 0]])Q, R = np.linalg.qr(A)print(\"Orthonormal basis (Q):\")print(Q)\n```\n\nThe Gram-Schmidt process is used to convert a set of vectors into an orthonormal basis. There are two basic concepts:\n\nThe Gram-Schmidt process accepts a set of linearly independent vectors and converts them into an orthonormal set.\n\nKey benefits of the Gram-Schmidt Process:\n\n1. Numerical Stability\n\n2. Dimensionality Reduction\n\n3. Effective Matrix Decomposition\n\n4. Feature Selection\n\n``` python\nimport numpy as npdef gram_schmidt(V):    \"\"\"    Classical Gram-Schmidt process.    Input:  V is a matrix whose columns are the vectors v1, v2, ..., vk    Output: Q is a matrix whose columns are the orthonormal basis u1, u2, ..., uk    \"\"\"    V = np.array(V, dtype=float)    n, k = V.shape    Q = np.zeros((n, k))    for i in range(k):        # Start with v_i        qi = V[:, i]        # Subtract projections onto previous q_j        for j in range(i):            proj = np.dot(Q[:, j], qi) * Q[:, j]            qi = qi - proj        # Normalize        Q[:, i] = qi / np.linalg.norm(qi)    return QV = np.array([    [1, 1],    [1, 0]])Q = gram_schmidt(V)print(\"Orthonormal basis (Classical GS):\")print(Q)\n```\n\n**Eigenvalues and Eigenvectors in PCA**\n\n``` python\nimport numpy as np# Example matrixA = np.array([    [4, 2],    [1, 3]])# Built-in functioneigenvalues, eigenvectors = np.linalg.eig(A)print(\"Eigenvalues:\")print(eigenvalues)print(\"\\nEigenvectors (columns):\")print(eigenvectors)# verify the eigen equationfor i in range(len(eigenvalues)):    v = eigenvectors[:, i]    λ = eigenvalues[i]    print(\"\\nCheck for eigenvalue:\", λ)    print(\"A @ v:\", A @ v)    print(\"λ * v:\", λ * v)\npython\nimport numpy as npA = np.array([[4, 1],              [2, 3]])# Step 1: eigenvalues & eigenvectorseigenvalues, eigenvectors = np.linalg.eig(A)# Step 2: Construct D (diagonal matrix)D = np.diag(eigenvalues)# Step 3: Matrix P (eigenvectors)P = eigenvectors# Step 4: Find P inverseP_inv = np.linalg.inv(P)# Step 5: Reconstruct A (should be equal to original A)A_reconstructed = P @ D @ P_invprint(\"Eigenvalues:\\n\", eigenvalues)print(\"Eigenvectors (P):\\n\", P)print(\"Diagonal matrix D:\\n\", D)print(\"P inverse:\\n\", P_inv)print(\"Reconstructed A:\\n\", A_reconstructed)\npython\nimport numpy as np# Symmetric matrixA = np.array([[2, 1, 0],              [1, 2, 1],              [0, 1, 2]])# Verify symmetryprint(\"Is symmetric:\", np.allclose(A, A.T))# Eigen decompositioneigvals, eigvecs = np.linalg.eigh(A)  # eigh is optimized for symmetric/hermitianprint(\"Eigenvalues:\", eigvals)print(\"Eigenvectors (columns):\\n\", eigvecs)# Verify orthogonality of eigenvectorsprint(\"Q^T Q:\\n\", eigvecs.T @ eigvecs)  # Should be identity\npython\nimport numpy as npimport matplotlib.pyplot as plt# Example: random 5x4 matrixX = np.random.randn(5, 4)# SVDU, S, VT = np.linalg.svd(X, full_matrices=False)print(\"U:\\n\", U)print(\"Singular values:\", S)print(\"V^T:\\n\", VT)# Rankrank = np.sum(S > 1e-10)print(\"Rank of X:\", rank)# Low-rank approximation (keep top 2 singular values)k = 2U_k = U[:, :k]S_k = np.diag(S[:k])VT_k = VT[:k, :]X_approx = U_k @ S_k @ VT_kprint(\"Low-rank approx:\\n\", X_approx)\n```\n\nNorms are directly built from inner products:\n\nWhy it’s important:\n\nThis pairs naturally with inner products since the two are mathematically linked.\n\nCosine similarity measures the similarity between two non-zero vectors by calculating the cosine of the angle between them. Similarity measure calculates the distance between data objects based on their feature dimensions in a dataset. A smaller distance indicates a higher similarity, while a larger distance indicates a lower similarity. **Cosine similarity** is a metric, helpful in determining how similar the data objects are, irrespective of their size.\n\nThis is equivalent to the matrix having all non-negative eigenvalues\n\nUnderstanding PSD matrices gives insight into:\n\n“Machine learning is nothing without mathematics” — I have heard a lot of them saying this. However, I truly understood it only when I deep dived into the machine learning algorithms. A solid understanding of linear algebra concepts helps to understand how machine learning works in a better and effective way.\n\nWith that, this blog comes to an end. This is only the beginning! I’m also planning to write detailed articles on **probability, statistics, and calculus for machine learning**, so stay tuned for more posts that will help you strengthen your mathematical foundation for ML.\n\nIf this series helped you, consider **following me **to get updates — there’s plenty more interesting stuff on the way.\n\n**Note: **Most of the content is represented as images due to the lack of LaTeX compatibility in Medium.\n\n[26 Important Concepts that you should learn in Linear Algebra for Machine Learning — PART II](https://pub.towardsai.net/26-important-concepts-that-you-should-learn-in-linear-algebra-for-machine-learning-part-ii-8110461624e3) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.", "url": "https://wpnews.pro/news/26-important-concepts-that-you-should-learn-in-linear-algebra-for-machine-part", "canonical_source": "https://pub.towardsai.net/26-important-concepts-that-you-should-learn-in-linear-algebra-for-machine-learning-part-ii-8110461624e3?source=rss----98111c9905da---4", "published_at": "2026-08-24 03:24:40+00:00", "updated_at": "2026-08-24 03:43:40.570086+00:00", "lang": "en", "topics": ["machine-learning"], "entities": ["Rajendran22", "Medium"], "alternates": {"html": "https://wpnews.pro/news/26-important-concepts-that-you-should-learn-in-linear-algebra-for-machine-part", "markdown": "https://wpnews.pro/news/26-important-concepts-that-you-should-learn-in-linear-algebra-for-machine-part.md", "text": "https://wpnews.pro/news/26-important-concepts-that-you-should-learn-in-linear-algebra-for-machine-part.txt", "jsonld": "https://wpnews.pro/news/26-important-concepts-that-you-should-learn-in-linear-algebra-for-machine-part.jsonld"}}