When you think of AI(Artificial Intelligence), what do you see? Most people, upon hearing that name, think of the recently developed LLMs(large language models): ChatGPT, Claude, Gemini, and many others. Such AI, used to create responses, writing, images, and much more, falls under the umbrella of generative AI. However, its counterpart, traditional AI, which has been around for close to 70 years, encompasses a variety of techniques, including rule-based systems and machine learning, that are designed to solve specific tasks rather than generate new content. Among the many approaches to traditional AI, supervised machine learning is one of the most widely used forms of machine learning. It learns from labeled examples to solve regression and classification problems. When a model employs regression, it is predicting a continuous numerical value/values after being given a set of inputs. On the other hand, a model can perform classification, where it predicts a category. Classification can be divided into binary classification, which has two possible classes, and multiclass classification, which has more than two classes.
So what exactly happens inside a model and what is it doing? Is it magic? A machine learning model is a mathematical function that takes an input to produce an output. Take, for example, a grade predictor where, given a student’s Hours Studied and Hours of Sleep, you are tasked with determining their exam score. To find the output, the following equation is used, where X is the input, b₁is the weight, and b₀ is the bias/y-intercept.
This process of taking an input and passing it through the model to produce a prediction is called a forward pass.
Feature — A single piece of information the model uses to make a prediction. Each feature represents one characteristic of an object or event. Denoted by an x + superscript, n.
Input — Known as x, it is the complete collection of features for one example. Instead of giving the model one feature at a time, all features are grouped into a single list or vector.
Target — Sometimes called the label or ground truth, the target is the correct answer that the model is trying to learn to predict during training.
Output/Prediction — Known as ŷ(y-hat), it is the model’s estimated output after processing an input vector. It represents the model’s best guess based on what it has learned from the training data.
Weight — A numerical value that determines how important a feature is when making a prediction. Every feature in the model has its own weight. Modified terms based on a feature can be introduced to add non-linearity(x1 becomes x1²).
Bias — An additional value that is added to the prediction after all of the weighted features have been combined. It allows the model to shift its prediction up or down instead of forcing it to pass through a fixed point.
Think of a machine learning model like a radio knob. When you try creating an ML model, it will be initialized with random weights. This would be a random channel number on your radio. Essentially, what the model wants to do is find the best combination of weights to reduce the difference between the estimated result(y-hat) and the actual result(y); in other words, it wants to find the right frequency that gives you a signal.
This is the basis of how the weights and bias are determined. After the model makes a prediction, it needs a way to measure how good or bad that prediction was. This measurement is called the loss (or cost) function. The larger the loss, the worse the prediction. During training, the model tries to minimize this value. Going back to the radio example, the cost function is the amount of static, which increases the farther you are from the channel and can be used to see how far you are from it. Essentially, the loss is the average of the difference between all the training data’s true values and the model’s predicted values. It is also important that positive and negative errors do not cancel each other out when calculating the overall loss.
Mean Squared Error(MSE) — After determining the difference between all predicted values and true labels, the results are squared before being averaged out. The purpose of this function is to punish larger errors more heavily than smaller errors. Because the errors are squared, an error of 4 contributes four times as much to the squared error as an error of 2. MSE also has the bonus of simplifying the gradient calculation process. The downside to this function is that it gravitates heavily toward outliers and struggles in datasets littered with them.
Mean Absolute Error(MAE) — This function aims to reduce the aforementioned outlier problem associated with MSE. By removing the square, the outlier no longer holds as much weight and allows the model to adjust much better. This is slightly more difficult to implement for gradient descent as the derivative changes over the domain.
Cross-entropy loss(CE)— CE is commonly used for classification tasks. In binary classification, the model generally produces a probability between 0 and 1. For example, a prediction of 0.9 could mean that the model estimates a 90% probability that an example belongs to the positive class. For binary classification, cross-entropy loss can be written as:
Cross-entropy heavily penalizes confident incorrect predictions.
For example, if the correct answer is 1 but the model predicts 0.99, the loss is very small. If the model predicts 0.01, the loss is very large because the model was extremely confident in the wrong answer. When understanding the loss function, a useful way of thinking about it is graphically. Imagine a mountain where the height represents the cost and the location (w_1, b) represents a position on the mountain. The model wants to find the location where the cost is lowest, or the “valley”.
Some machine learning problems have relatively simple loss surfaces culminating at one minimum. This is called a convex function(left graph). On the other hand, non-convex loss landscapes(right graph), native to many neural networks, contain many curves, valleys, and potential local minima.
The model therefore needs a method for determining which direction to move.
This is where gradients come in.
The gradient describes how the loss changes with respect to each parameter. It tells the model the direction in which the loss increases most rapidly and how quickly it changes in that direction. To calculate the gradient, the model finds the partial derivative of the loss with respect to each weight and bias in a process called backpropagation. In neural networks, backpropagation is the process of efficiently calculating these gradients by working backward through the network, starting from the output and determining how much each parameter contributed to the final loss.
Gradient Descent, on the other hand, is the method of tweaking weights to arrive at a lower loss region. After backpropagation, gradient descent multiplies the gradient by the learning rate(explained later) and accordingly updates the weights.
Another form of gradient descent, called Stochastic Gradient Descent (SGD), updates the model’s weights after processing each training example (batch size = 1). Because it only processes one example at a time, SGD requires less memory and is better suited for larger datasets. However, its updates are more noisy or “bouncy” because each update is based on only one example, which can make the path toward the minimum less stable and require more updates to converge.
After the inputs are multiplied by their weights and the bias is added, the resulting value is passed through a mathematical function called an activation function**.** Activation functions serve several purposes. Some of these include introducing non**-** linearity into a neural network, allowing it to learn complex relationships that a simple linear model cannot. Additionally, the output of the activation function can then be passed to the next layer of the network. Different activation functions serve different purposes; for example, ReLU is commonly used in hidden layers, while sigmoid is often used in the output layer for binary classification. In a neural network, this process occurs repeatedly across multiple layers. The outputs of one layer become the inputs to the next, allowing the network to gradually transform the original input into a final prediction.
When creating a model, there are certain parameters you need to choose prior to creating the model. Here are a few:
Learning Rate — Used in gradient descent, the learning rate decides how much the model will change its weights after calculating the gradient. A larger learning rate causes the model to take larger steps toward the minimum of the loss function. However, if the learning rate is too large, the model may overshoot the minimum and fail to converge. A learning rate that is too small can make training extremely slow.
Batch size — This is how much training data will be processed before updating its weights. For example, in a dataset with 1,000 training examples where the batch size is 100, the model will process 100 samples and update its weights before moving to the next 100 samples.
Epochs — This is one complete pass through the training dataset. Using the previous example dataset, if a model is trained for 10 epochs, then it will go through all 1000 examples 10 times. Each epoch is divided into batches when a batch size smaller than the entire dataset is used. Too few epochs can cause underfitting, while too many can lead to overfitting, both resulting in poor generalization(performance) to new data.
Neurons — A neuron is the smallest basic unit in a Neural Network(NN). It takes an input, multiplies it by weights, adds the bias term, and then usually passes it through an activation function. The result will then either be sent to the next layer or be used to produce the model’s final output.
Layers — A layer is a group of neurons that processes data at the same stage of an NN. Each neuron in a layer receives one or more inputs from the previous layer, performs its calculations, and passes its output. There are multiple types of layers, and a normal NN will possess an input layer, one or more hidden layers, and an output layer. While additional layers may aid a model in learning more complex problems, too many can cause the vanishing gradient problem, where earlier weights become less important.
Hyperparameters are settings chosen before or around training, while parameters such as weights and biases are learned by the model during training.
The Internal Workings of a Machine Learning Model was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.