Gradient Descent

A developer refactoring a gradient descent tutorial found that replacing a fixed-iteration loop with an early-stopping convergence check and switching to NumPy vectorized operations dramatically improved performance, but the biggest gain came from adding Z-score feature scaling, which cut iterations from 10,000 to 120.

Gradient Descent The core issue is that the lab code just loops for a set number of epochs. In a real-world AI workflow, this is inefficient. If the model converges in 500 iterations, you're wasting 9,500 cycles of compute. Conversely, if the learning rate is too low, 10,000 iterations might not even get you close to the global minimum. The "Fixed Iteration" Problem vs. Convergence Criteria The standard approach in these tutorials is: for i in range iterations : update weights calculate cost But the mathematically sound way is to monitor the change in the cost function. If the difference between the cost at $t$ and $t-1$ falls below a certain threshold epsilon , the model has converged. I tried to refactor the lab code to implement an early-stopping mechanism. Here is the logic I'm using to replace the fixed loop: python def compute gradient descent X, y, w, b, learning rate, epsilon=1e-6, max iters=10000 : prev cost = float 'inf' for i in range max iters : Gradient calculation grad w, grad b = compute gradient X, y, w, b Update parameters w = w - learning rate grad w b = b - learning rate grad b Convergence check current cost = compute cost X, y, w, b if abs prev cost - current cost < epsilon: print f"Converged at iteration {i}" break prev cost = current cost return w, b Diagnosis of Inefficiency Even with the convergence check, I noticed the performance was abysmal compared to a vectorized implementation. The lab often encourages a "beginner-friendly" loop-based approach to help you understand the math, but it's a nightmare for actual performance. Specifically, the original implementation does this for the gradient: Inefficient loop-based gradient dj dw = 0 for i in range m : dj dw i = f wb - y i X i dj dw += dj dw i dj dw = dj dw / m When I shifted to a NumPy vectorized approach, the execution time dropped significantly because I eliminated the Python-level loop. For anyone doing a deep dive into this, use dot products: Vectorized version errors = np.dot X, w + b - y dj dw = 1/m np.dot X.T, errors dj db = 1/m np.sum errors The Remaining Bug: Vanishing/Exploding Gradients The "inefficiency" I felt wasn't just about the code speed—it was about the behavior. Without feature scaling, my manual implementation was oscillating wildly or taking forever to converge, even with the epsilon check. I realized the lab doesn't emphasize enough that if your features are on different scales e.g., one feature is 0-1 and another is 0-1000 , the cost function becomes a very elongated "bowl," making gradient descent struggle. To fix this, I added Z-score normalization before passing the data into the descent function: X scaled = X - np.mean X, axis=0 / np.std X, axis=0 This small change made the model converge in 120 iterations instead of timing out at 10,000. If you're struggling with a manual implementation that feels "off," check your feature scaling before blaming the learning rate. Next Attie: Turning Bluesky Data into Research → /en/threads/2797/