Lesson 7: Understanding the Role of Bias, Parameters and Hyperparameters in Machine Learning

A machine learning tutorial explains that traditional machine learning models saturate because they reach theoretical limits and cannot extract more information from additional data, while large neural networks with 20–25 hidden layers overcome this by learning complex patterns. The lesson also defines bias as a parameter that shifts the activation function, noting that extreme bias values can cause issues such as outputs near zero or large positive values.

Understanding the Role of Bias, Parameters and Hyperparameters in Machine Learning Let’s discuss what is Traditional Machine Learning, Small Neural Network & Large Neural Network. Traditional Machine Learning: Traditional machine learning relies on hand-crafted features and algorithms to detect patterns in data. It requires large amounts of data to achieve high performance, and after a point of time its performance begins to saturate. Small Neural Network: Small neural networks typically contain 1–2hidden layers and are used to identify small patterns and relations in data. They require less data than traditional machine learning and can achieve high accuracy with smaller datasets. Large Neural Network: Large neural networks contain 20–25hidden layers, which enable them to detect complex patterns and relationships in large datasets. These networks require significant computational resources and training time but can achieve much higher performance than traditional machine learning or small neural networks. Q. In the Traditional Machine Learning, after a point of time the model starts to saturate — Why? Answer: The graph of data vs performanceis representative of the fact that models initially start performing better as more data is added to the training set. This is because the model is able to better generalize its parameters from the larger data set and thus the performance of the model increases. However, after a point of time the Traditional Machine Learning model starts to saturate, meaning that no matter how much data is added to the training set the performance of the model does not get any better. This is because the model has reached its theoretical limits and no more information can be gained from the additional data. Thus, after the model has reached its saturation point the performance of the model starts to degrade as more data is added. Saturated Model: A saturated model is a model that has been over-fitted to the data, meaning it contains too many parameters and is unable to accurately capture the underlying trends of the data. This can lead to a model that is too complex and not able to generalize to new data, resulting in poor predictions. This leads to extremely high-variancepredictors that are more sensitive to the noise than the true signal in the data, making them unreliable for predictions. Q. How Large Neural Network solve this saturation problem of Traditional Machine Learning models? Large Neural networks are able to overcome the saturation problem faced by traditional machine learning algorithms by adding more parameters and layers to the model. By doing so, the model is able to learn more complex patterns and thus can more accurately fit the data. This allows the model to continue to improve even after the traditional machine learning algorithm has reached its saturation point. Additionally, neural networks also allow for more efficient training of the model by utilizing techniques such as batch normalization and weight decay, which help to reduce overfitting and improve the generalization of the model. Bias in Machine Learning Bias: Bias is a parameter in a machine learning algorithm that allows the activation function to be shifted to the left or right. Bias is used to adjust the output values of the model and influences how easy it is for a node to fire. Positive values of bias tend to produce higher output values while negative values of bias tend to produce lower output values. Role of Bias: Bias allows the activation function to better fit the data. Bias value is used to shift the entire sigmoid curve. Bias does not interact with the actual input data. Bias determines how easy it is to get a node to fire. Large bias values lead to large positive outputs. Negative bias values lead to sigmoid outputs near to 0. Small bias values depend entirely on the values of weights and inputs. Q. What problem may arise if the bias is very big or very small? Answer: The equation y=mx+b is a linear model that describes how a dependent variable y is related to an independent variable x . The slope m of the line describes the relationship between x and y, while the intercept or bias b is the point where the line crosses the y-axis. Depending on the value of bias b , the model can give priority to either the features or the bias. If y=mx<b; then the model will depend on the bias. The model will give priority to the bias. If y=mx b; then the features will get priority rather than bias. If the bias is very big, the model may be overly sensitive to the data and could result in inaccurate predictions. Overly sensitive data can lead to models that are overly complex, resulting in overfitting. This means that the model is too closely tailored to the training data and performs poorly on unseen data. On the other hand, if the bias is very small, the model may be less sensitive to the data and could result in predictions that are too general. If a model is not sensitive enough to data, it can cause underfitting. This means that the model is not able to learn from the data and therefore unable to make accurate predictions. In both cases, the model could be less accurate or biased. So it is important to have a medium bias in order to ensure that the model is accurate and unbiased. Because having a medium bias in a linear model ensures that the model is neither too sensitive nor too general with regard to the data. This means that the model is more likely to make accurate and unbiased predictions. Parameters & Hyperparameters Parameters are values that are updated during the training process, they are updated by the algorithm, while hyperparameters are values that are set before training a model. Parameters: Parameters are internal to the model and whose value can be estimated from data.Parameters are required by the model when making predictions and are often not set manually by the practitioner. Examples of parameters include — Weight matrix of a model: A weight matrix is a set of numerical values used to represent the strength of the connections between nodes in a neural network. Coefficients in a Linear Regression:Coefficients in a linear regression are numerical values that determine the strength of the relationship between an independent variable and the dependent variable. Depth of a Decision Tree: The depth of a decision tree is the number of levels or layers of nodes present in the tree, which determines how complex the model is. Hyperparameters: Hyperparameters are external to the model and whose value cannot be estimated from data. They are set before training and determine how the network is trained. Examples of hyperparameters include — Dropout: A technique used to reduce overfitting in neural networks by randomly disabling certain neurons during training. Number of hidden layers: The number of layers between the input and output layers of a neural network. Weight initialization: The process of setting the initial values for weights in a neural network. Active function: An activation function is an non-linear transformation applied to the input of a neuron to produce an output. Epoch: An epoch is a single forward and backward pass of the entire training dataset. Batch size: The number of training samples used in one forward/backward pass. Optimizer: The process of adjusting the weights of a neural network to minimize a cost function. Learning rate: A parameter that controls how quickly a neural network learns. Loss function: A function that measures the performance of a neural network. In short, what we assign in the model are called — hyperparameters. While what we don’t assign in the model but continuously update are called — parameters. Difference between Parameters and Hyper-parameters The main differences between parameters and hyperparameters are: Parameters are internal to the model and whose value can be estimated from data; whereas hyperparameters are external to the model and whose value cannot be estimated from data. Parameters are required by the model when making predictions; whereas hyperparameters are set before training and determine how the network is trained. Parameters are often not set manually by the practitioner; whereas hyperparameters are set manually by the practitioner. Parameters are often saved as part of the learned model; whereas hyperparameters are not saved as part of the learned model. Parameters define the skill of the model on a problem; whereas hyperparameters determine the network structure and used in processes to help estimate model parameters. Hyperparameters of Traditional Machine Learning Models Hyper-parameter Tuning is the process of selecting the best set of hyper-parameters for a given Machine Learning algorithm. It is done by testing different combinations of hyper-parameters and using techniques such as Random Search, Grid Search, Bayesian Optimizer, and Gradient-based to find the best combination. Here, after hyper-parametertuning the neural network is now more optimized and efficient. Dropout is used in the neural network in order to disable certain neurons. Dropout also reduced the chance of overfitting. Hyper-parameter tuning can be used to adjust the structure of the network, such as the number of layers, the number of nodes per layer, the activation functions used in each layer, and the type of regularization used. By optimizing these parameters, the neural network can be tuned to better fit the data, resulting in improved performance. Hyper-parameter Tuning Techniques: Random Search:Random Search randomly chooses combinations of hyperparameters from searchable space all possible combinations . An example of Random Search is randomly selecting the number of layers, neurons per layer, and learning rate. Grid Search:Grid Search tests all possible combinations of hyperparameters of given Machine Learning algorithm. An example of Grid Search is testing different combinations of layers, neurons per layer, and learning rate. Bayesian Optimizer:Bayesian optimizer is based upon Bayes Rule and considers previously known knowledge to help narrow down the search space of good hyperparameter combinations. An example of Bayesian optimizer is using Bayesian optimization to find the best combination of learning rate, number of layers and number of neurons per layer. Gradient-based:Gradient-based is a methodology to optimize several hyperparameters, based on the computation of the gradient of a machine learning model selection criterion with respect to the hyperparameters. An example of Gradient-based is using gradient descent to find the optimal combination of learning rate, number of layers and number of neurons per layer. Q. Why do we need hyper-parameter tuning? Answer: Because of the following reasons, we need hyper-parameter tuning — Finding the right balance between bias and variance to ensure high accuracy and good generalization. Preventing vanishing/exploding gradient problem by tweaking hyper-parameters like learning rate, activation function, and number of layers. Overcoming saddle points and local optima by adjusting learning rate and choosing appropriate optimizers. Avoiding no convergence by using adaptive learning rate optimizers like Adam or using decay in optimizers. Avoiding sigmoid and tanh activation function problems by limiting their usage to shallow networks. Speeding up learning by getting the right set of hyper-parameters. Dealing with umpteen combinations of all the hyper-parameters. Achieving more from less by making the best out of current architecture and slowly increasing complexity. Summary: Traditional Machine Learning relies on hand-crafted features and algorithms to detect patterns in data. Small Neural Networks contain 1–2 hidden layers and are used to identify small patterns and relations in data. Large Neural Networks contain 20–25 hidden layers, which enable them to detect complex patterns and relationships in large datasets. Traditional Machine Learning models start to saturate after a point of time due to reaching its theoretical limits. Saturated models lead to extremely high-variance predictors that are being pushed around by the noise more than the actual data. Large Neural Networks are able to overcome the saturation problem by adding more parameters and layers, allowing the model to learn more complex patterns and thus improve its accuracy. Bias is a parameter in a machine learning algorithm that allows the activation function to be shifted to the left or right. Bias allows the activation function to better fit the data. If the bias is very big, the model may be overly sensitive to the data and could result in inaccurate predictions. If the bias is very small, the model may be less sensitive to the data and could result in predictions that are too general. It is important to have a medium bias in order to ensure that the model is accurate and unbiased. Parameters are internal values of a model that are estimated from data, while hyperparameters are external values that need to be set before training a model. Parameters are used to make predictions, while hyperparameters determine how the model is trained. Hyperparameter tuning is the process of finding the best set of hyperparameters for a given Machine Learning algorithm. Techniques used for hyperparameter tuning include Random Search, Grid Search, Bayesian Optimizer, and Gradient-based. Hyperparameters are manually set by the practitioner, while parameters are automatically updated during training. See you in the next Article. That’s it for now. If you found this article interesting, helpful and if you learn something from this article, please follow, comment and leave feedback. If you want to connect with me, here is my LinkedIn.