AlexNet: Pioneering the Path to Modern Deep Learning

In 2012, Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton published the AlexNet paper at NIPS, introducing a deep convolutional neural network architecture that revolutionized computer vision by winning the ImageNet competition. The architecture, which uses multiple convolutional layers, ReLU activations, and dropout, laid the foundation for modern deep learning and remains influential today.

In the preceding article https://patilswaraj22.medium.com/keras-based-implementation-of-lenet-5-db678b11c61a , we explored the utilization of the LeNet-5 Architecture and the remarkable capability of Convolutional Neural Networks in recognizing handwritten characters within images. Yann Le Cunn introduced the LeNet-5 architecture in 1998 , a time when computing resources with significant power were limited, resulting in a limited range of architectural modifications. In the year 2012, there was a complete revolution observed in this field. A paper was published at the NIPS Neural Information Processing Systems conference named “ ImageNet Classification with Deep Convolutional Neural Networks ,” authored by In comparison with LeNet, AlexNet is a complex architecture that uses different Kernels of various sizes. Let’s take a look at the architecture below. Initially, this architecture might look terrifying, but a simplified or easy-to-understand image can be seen below. Notable Points These were the most important points discussed in the paper, which are still used today. P.S. -: The input size mentioned in the paper is 224 x 224 x 3, and the first convolutional layers output consists of 96 Kernels with size 11 x 11 along with stride = 3. So according to this, the output should be 224–11 /4 + 1 X 224 -11 /4 + 1 = 54 x 54 This does not match the output shown in the architecture, which is 55 x 55 . The possible explanation for this is that the author might have performed a padding operation on the image to increase the size of the image from 224 x 224 to 227 x 227. Now, if we consider the input image size to be 227 x 227, then the entire calculation falls right into place. 227–11 /4 + 1 X 227 -11 /4 + 1 = 55 x 55 Now that we have gone through the crucial points let’s look at the architecture implementation of the same. The entire Python code for the same, along with the Jupyter Notebook, can be accessed here: python import numpy as npimport pandas as pdimport seaborn as snsimport matplotlib.pyplot as pltimport tensorflow as tffrom tensorflow.keras import layersfrom tensorflow.keras.datasets import mnistfrom tensorflow.keras.layers import Dense, Flatten, Input, Dropoutfrom tensorflow.keras.layers import Conv2D, MaxPooling2Dfrom tensorflow.keras.models import Modelfrom tensorflow.keras.callbacks import ModelCheckpoint, EarlyStoppingfrom sklearn.metrics import confusion matrix Configurable ParametersEPOCHS = 10BATCH SIZE = 32LOSS = tf.keras.metrics.categorical crossentropyOPTIMIZER = tf.keras.optimizers.SGD METRIC = 'accuracy' def plot errors history, epochs : Plotting Train and Validation Loss epochs range = list range 1, epochs + 1 train loss = history.history 'loss' val loss = history.history 'val loss' plt.figure 1, figsize= 10, 6 plt.plot epochs range, train loss, label='train loss' plt.plot epochs range, val loss, label='validation loss' plt.xlabel 'Epochs' plt.ylabel 'Categorical Cross Entropy' plt.legend plt.show def plot confusion matrix y test, y pred, classes= : cm = confusion matrix y test, y pred, labels=classes cm df = pd.DataFrame cm, index=classes, columns=classes plt.figure 1, figsize= 16, 8 sns.set font scale=1.5, color codes=True, palette='deep' sns.heatmap cm df, annot=True, annot kws={'size': 16}, fmt='d', cmap='YlGnBu' plt.ylabel "True Label" plt.xlabel "Predicted Label" plt.title 'Confusion Matrix' plt.show def model training model, X train, y train, X cv, y cv, epochs, batch size,loss, optimizer, metrics : Initialise optimizers model.compile loss=loss, optimizer=optimizer, metrics=metrics Enabling Early Stopping es = EarlyStopping monitor='val loss', mode='min', verbose=1, patience=3 Enabling check point mc = ModelCheckpoint filepath='bestModel.h5', monitor='val acc', mode='max', verbose=1, save best only=True Model fitting history = model.fit X train, y train, validation data= X cv, y cv , epochs=epochs, batch size=batch size, verbose=1, callbacks= es, mc return model, historyclass AlexNet: def init self, input shape, output shape : self.input shape = input shape self.output shape = output shape def model initialise self : Define the AlexNet Architecture ip = Input shape= self.input shape 1st Convolutional Layer followed by Response Normalised and Pooling x = Conv2D filters=96, kernel size= 11,11 , padding="same", strides= 4,4 , activation ="relu" ip x = layers.Lambda tf.nn.local response normalization x x = MaxPooling2D pool size = 3,3 , padding="same", strides= 2,2 x 2nd Convolutional Layer followed by Response Normalised and Pooling x = Conv2D filters=256, kernel size= 5,5 , padding="same", strides= 2,2 , activation = "relu" x x = layers.Lambda tf.nn.local response normalization x x = MaxPooling2D pool size = 3,3 , padding="same", strides= 2,2 x 3rd Convolutional Layer x = Conv2D filters=384, kernel size= 3,3 , padding="same", strides= 1,1 , activation = "relu" x 4th Convolutional Layer x = Conv2D filters=384, kernel size= 3,3 , padding="same", strides= 1,1 , activation = "relu" x 5th Convolutional Layer followed by Response Normalised and Pooling x = Conv2D filters=256, kernel size= 3,3 , padding="same", strides= 1,1 , activation = "relu" x x = layers.Lambda tf.nn.local response normalization x x = MaxPooling2D pool size = 3,3 , padding="same", strides= 2,2 x Flattening x = Flatten x 1st Fully Connected Layer with DropOuts x = Dense units=4096, activation="relu" x x = Dropout 0.5 x 2nd Fully Connected Layer with DropOuts x = Dense units=4096, activation="relu" x x = Dropout 0.5 x Output Layer op = Dense units=self.output shape, activation="softmax" x Define Model model = Model inputs=ip, outputs=op model.summary return modelif name == ' main ': Loading the dataset and perform splitting X train, y train , X test, y test = mnist.load data Resize image to 32x 32 X train = np.array np.pad X train i , pad width=2 for i in range X train.shape 0 X test = np.array np.pad X test i , pad width=2 for i in range X test.shape 0 Performing reshaping operation X train = X train.reshape X train.shape 0 , 32, 32, 1 X test = X test.reshape X test.shape 0 , 32, 32, 1 Normalization X train = X train / 255 X test = X test / 255 One Hot Encoding y train = tf.keras.utils.to categorical y train, 10 y test = tf.keras.utils.to categorical y test, 10 Define image size and number of classes image size = X train.shape 1: classes = y train.shape 1 print f"Size of each image = {image size}" print f"Nos of Classes = {classes}" Create model instance and initialise Lenet Model alexnet = AlexNet image size, classes model = alexnet.model initialise epochs = EPOCHS batch size = BATCH SIZE loss = LOSS optimizer = OPTIMIZER metric = METRIC model, history = model training model, X train, y train, X test, y test, epochs, batch size, loss, optimizer, metric Plot trainning and test error plot errors history, min epochs, len history.history 'loss' Perform Prediction y pred = model.predict X test Get list of prediction y pred = np.argmax y pred, axis=1 y test = np.argmax y test, axis=1 Show Confusion Matrix plot confusion matrix y test, y pred, list range classes The entire Python code for the same, along with the Jupyter Notebook, can be accessed here: References : — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — If this was useful, consider giving it a clap, it really helps. I write about ML, AI, and technology. Follow me here on Medium so you don’t miss the next one. 📌 More from me: → Keras Implementation of LE-NET AlexNet: Pioneering the Path to Modern Deep Learning https://pub.towardsai.net/alexnet-pioneering-the-path-to-modern-deep-learning-16525572e266 was originally published in Towards AI https://pub.towardsai.net on Medium, where people are continuing the conversation by highlighting and responding to this story.