Keras Based Implementation of LeNet-5

A developer published a Keras-based implementation of the LeNet-5 convolutional neural network architecture, originally proposed by Yann LeCun in 1989 for handwritten digit recognition. The implementation uses functional programming in Keras to replicate the original architecture, which includes convolutional, pooling, and fully connected layers.

Introduction LeNet-5 is one of the first Convolutional Neural Network models proposed by Yann LeCun et al. in 1989. The architecture was first observed in the paper “ Gradient-Based Learning Applied to Document Recognition .” The authors have used this architecture to recognize handwritten digits. We won’t be going into the details of this paper; rather, we will focus more on implementing the architecture in Keras. The LeNet-5 architecture has described in the paper is shown below LeNet 5 Architecture Architecture Overview The input image that is feed to the network is 32 x 32 . This is fed to the first convolutional layer with a filter size of 5 x 5 , stride =1, and the image is fed to 6 such filters filters are also called Kernels , which generates an output C1 . Kernels are nothing but a feature extractor of an image, and the extracted features are also known as “ feature maps .” The size of each feature map here is 28 x 28, and 6 such feature maps are generated. Let’s understand how we obtained the dimensions 28 x 28 . This is one of the most important things in the understanding of architecture. Let us assume the size of image be n x n and the kernel size be k x k and stride be s So the output size of the feature maps is obtained by the formula n-k / s + 1 x n-k / s + 1 . In our case the value of n = 32 and k = 5 so the feature map size 32–5+1 x 32–5+1 = 28 x 28. Now that we have understood this, the rest of the architecture is based on the above calculation. The second layer is the pooling layer or subsampling layer. The subsampling layer performs sampling of the feature map to reduce its size. This is done to reduce the overall computation since the neighbouring pixels store more or less similar information. The subsampling layer used is of size 2 x 2 with stride = 2 . The output of the subsampling layer is the feature maps S2 with size 14 x 14 Calculation is the same as the above . The third layer again is the convolutional layer with filter size 5 x 5 and stride = 1 . The only difference here is we would be the number of kernels 16 kernels. The output C3 would be of the size 10 x 10 with 16 feature maps. “ Layer C3 is a convolutional layer with 16 feature maps. Each unit in each feature map is connected to several 5 x 5 neighbourhoods at identical locations in a subset of S2’s feature maps. The first six C3feature maps take inputs from every contiguous subset of three feature maps in S2 The next six take input from every contiguous subset of four The next three take input from some discontinuous subsets of four. Finally, the last one takes input from all S2 feature maps .” Each column indicates which units combine feature map in S2 in a particular feature map of C3. Mapping of each S2 on C3 The obtained C3 consists of 16 layers, each of 10 x 10. As earlier, again subsampling layer here is used of size 2 x 2 with stride = 2 . The output here is a feature map S4 with 16 layers of size 5 x 5. The feature map S4 is convoluted with 120 kernels each of size 5 x 5 followed by a flattening layer to get the output C5 layer. A fully connected layer F6 of 84 layers is connected after the C5 layers. At last, we have 10 output layers obtained from the F6 layer as the target variable has 10 distinct values. Now let's look at the implementation of the same architecture in Keras. Implementation Below is the implementation of the entire architecture in Keras using Functional programming. Importsimport numpy as npimport pandas as pdimport seaborn as snsimport matplotlib.pyplot as pltimport kerasfrom keras.datasets import mnistfrom keras.layers import Dense, Flatten, Inputfrom keras.layers import Conv2D, MaxPooling2Dfrom keras.models import Sequential, Modelfrom keras.callbacks import ModelCheckpoint, EarlyStoppingfrom sklearn.metrics import confusion matrix Configurable ParametersEPOCHS = 20BATCH SIZE = 32LOSS = keras.metrics.categorical crossentropyOPTIMIZER = keras.optimizers.SGD METRIC = 'accuracy' def plot errors history, epochs : Plotting Train and Validation Loss epochs range = list range 1, epochs + 1 train loss = history.history 'loss' val loss = history.history 'val loss' plt.figure 1, figsize= 10, 6 plt.plot epochs range, train loss, label='train loss' plt.plot epochs range, val loss, label='validation loss' plt.xlabel 'Epochs' plt.ylabel 'Binary Cross Entropy' plt.legend plt.show def plot confusion matrix y test, y pred, classes= : cm = confusion matrix y test, y pred, labels=classes cm df = pd.DataFrame cm, index=classes, columns=classes plt.figure 1, figsize= 16, 8 sns.set font scale=1.5, color codes=True, palette='deep' sns.heatmap cm df, annot=True, annot kws={'size': 16}, fmt='d', cmap='YlGnBu' plt.ylabel "True Label" plt.xlabel "Predicted Label" plt.title 'Confusion Matrix' plt.show def model training model, X train, y train, X cv, y cv, epochs, batch size,loss, optimizer, metrics : Initialise optimizers model.compile loss=loss, optimizer=optimizer, metrics=metrics Enabling Early Stopping es = EarlyStopping monitor='val loss', mode='min', verbose=1, patience=3 Enabling check point mc = ModelCheckpoint filepath='bestModel.h5', monitor='val acc', mode='max', verbose=1, save best only=True Model fitting history = model.fit X train, y train, validation data= X cv, y cv , epochs=epochs, batch size=batch size, verbose=1, callbacks= es, mc return model, historyclass Lenet5: def init self, input shape, output shape : self.input shape = input shape self.output shape = output shape def model init self, kernel size= 5, 5 , pool size= 2, 2 : if isinstance kernel size, tuple and isinstance pool size, tuple : Define the LeNet-5 architecture ip = Input shape=self.input shape 1st Convolutional Layer followed by Max pooling 6 kernels x = Conv2D filters=6, kernel size=kernel size, activation='tanh' ip x = MaxPooling2D pool size=pool size x 2nd Convolution Layer followed by Max pooling 16 kernels x = Conv2D filters=16, kernel size=kernel size, activation='tanh' x x = MaxPooling2D pool size=pool size x 3rd Convolution Layer followed by Flattening 120 Kernels x = Conv2D filters=120, kernel size=kernel size, activation='tanh' x x = Flatten x Fully Connected Dense Layer x = Dense units=84, activation='tanh' x Final Output Softmax layer op = Dense units=self.output shape, activation='softmax' x Define Model model = Model inputs=ip, outputs=op model.summary return modelif name == ' main ': Loading the dataset and perform splitting X train, y train , X test, y test = mnist.load data Resize image to 32x 32 X train = np.array np.pad X train i , pad width=2 for i in range X train.shape 0 X test = np.array np.pad X test i , pad width=2 for i in range X test.shape 0 Performing reshaping operation X train = X train.reshape X train.shape 0 , 32, 32, 1 X test = X test.reshape X test.shape 0 , 32, 32, 1 Normalization X train = X train / 255 X test = X test / 255 One Hot Encoding y train = keras.utils.to categorical y train, 10 y test = keras.utils.to categorical y test, 10 Define image size and number of classes image size = X train.shape 1: classes = y train.shape 1 Create model instance and initialise Lenet Model lenet = Lenet5 image size, classes model = lenet.model init epochs = EPOCHS batch size = BATCH SIZE loss = LOSS optimizer = OPTIMIZER metric = METRIC model, history = model training model, X train, y train, X test, y test, epochs, batch size, loss, optimizer, metric Plot trainning and test error plot errors history, epochs Perform Prediction y pred = model.predict X test Get list of prediction y pred = np.argmax y pred, axis=1 y test = np.argmax y test, axis=1 Show Confusion Matrix plot confusion matrix y test, y pred, list range classes Model Summary Variations of train and validation loss on different epochs Error Plots Confusion Matrix of all the classes The entire python code for the same, along with the Jupyter notebook, can be accessed here: SP2203/Lenet5 References: http://yann.lecun.com/exdb/publis/pdf/lecun-01a.pdf https://keras.io/api/ — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — If this was useful, consider giving it a clap, it really helps. I write about ML, AI, and technology. Follow me here on Medium so you don’t miss the next one. 📌 More from me: → Smoothening noisy GNSS data Keras Based Implementation of LeNet-5 was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.