The core issue is that the lab code just loops for a set number of epochs. In a real-world AI workflow, this is inefficient. If the model converges in 500 iterations, you're wasting 9,500 cycles of compute. Conversely, if the learning rate is too low, 10,000 iterations might not even get you close to the global minimum.
The "Fixed Iteration" Problem vs. Convergence Criteria #
The standard approach in these tutorials is:
for i in range(iterations):
But the mathematically sound way is to monitor the change in the cost function. If the difference between the cost at $t$ and $t-1$ falls below a certain threshold (epsilon), the model has converged.
I tried to refactor the lab code to implement an early-stopping mechanism. Here is the logic I'm using to replace the fixed loop:
def compute_gradient_descent(X, y, w, b, learning_rate, epsilon=1e-6, max_iters=10000):
prev_cost = float('inf')
for i in range(max_iters):
grad_w, grad_b = compute_gradient(X, y, w, b)
w = w - learning_rate * grad_w
b = b - learning_rate * grad_b
current_cost = compute_cost(X, y, w, b)
if abs(prev_cost - current_cost) < epsilon:
print(f"Converged at iteration {i}")
break
prev_cost = current_cost
return w, b
Diagnosis of Inefficiency #
Even with the convergence check, I noticed the performance was abysmal compared to a vectorized implementation. The lab often encourages a "beginner-friendly" loop-based approach to help you understand the math, but it's a nightmare for actual performance.
Specifically, the original implementation does this for the gradient:
dj_dw = 0
for i in range(m):
dj_dw_i = (f_wb - y[i]) * X[i]
dj_dw += dj_dw_i
dj_dw = dj_dw / m
When I shifted to a NumPy vectorized approach, the execution time dropped significantly because I eliminated the Python-level loop. For anyone doing a deep dive into this, use dot products:
errors = np.dot(X, w) + b - y
dj_dw = (1/m) * np.dot(X.T, errors)
dj_db = (1/m) * np.sum(errors)
The Remaining Bug: Vanishing/Exploding Gradients #
The "inefficiency" I felt wasn't just about the code speed—it was about the behavior. Without feature scaling, my manual implementation was oscillating wildly or taking forever to converge, even with the epsilon
check. I realized the lab doesn't emphasize enough that if your features are on different scales (e.g., one feature is 0-1 and another is 0-1000), the cost function becomes a very elongated "bowl," making gradient descent struggle.
To fix this, I added Z-score normalization before passing the data into the descent function:
X_scaled = (X - np.mean(X, axis=0)) / np.std(X, axis=0)
This small change made the model converge in 120 iterations instead of timing out at 10,000. If you're struggling with a manual implementation that feels "off," check your feature scaling before blaming the learning rate.
Next Attie: Turning Bluesky Data into Research →