# Eye of the Infra — What is a batch and an epoch? > Source: > Published: 2026-09-01 05:09:01+00:00 Firstly, In the previous [article](https://saikaushikponnekanti.substack.com/p/eye-of-the-infra-training-vs-inference), we founded a startup whose mission is to stop people from losing their life savings to fraudulent emails.. One thing we left out is that we forgot to name the startup. I think I understand why, we are engineers at heart and want to get our hands dirty as quickly as possible and as such started thinking about the product. Now, some customers are expressing interest and we have to create a brand. We came up with a quirky, funny and imaginative name “Phish & Chips” (as our startup identifies spam emails. Really cool, right!). Let’s take a moment here and congratulate ourselves on taking the first step to creating the brand. Now let’s get back to the topic at hand. We started off the company with a dataset consisting of 100,000 emails which have already been pre-classified as “Spam” and “Not Spam”. Now in the previous article, we were talking about what training is Give the model some data → let it make a prediction → measure how wrong it is → adjust the weights → repeat. But we didn’t ask the question. “How do we give the 100,000 emails we have to the model?” First, when you think about it, it’s not such a big or great question. I mean u can think how does it matter right? But as you are going to see, it matters a whole lot when it comes to the infrastructure layer. Let’s get started. The first thing that comes to mind is why not directly dump the 100,000 emails directly to the trainer and process them in one shot. You wouldn’t be wrong. So the pipeline looks something like this. This looks decent I must say. For any company to be successful, we need to try to break down our own ideas to see if there are any problems. We do have faith in our product and our company and we expect that we are going to be successful. Once the customer base starts growing, our 100,000 emails will also grow. So tomorrow it may grow to 10 Million and frankly even billions in the future. So what problems are we going to face by dumping the total training dataset to the model? Our GPUs only have a finite amount of memory and frankly, the training data is not the only one using that memory. The model itself takes some memory and during the training, we need memory for intermediate results, gradients etc. Processing the entire dataset in one shot won’t scale as the dataset grows. The immediate next thing that comes to mind is what happens if we try and process 1 at a time. You might think this might be an overkill and you are right but let’s run through the option first. So, the pipeline looks like this. Then we do Email 2, then Email 3, so on until 100,000. Let’s try and see what the problems with this approach are. GPUs are very good at doing lots of similar computations in parallel. In this approach, we are literally feeding GPU 1 email at a time. We are not giving the GPUs enough work. So, we are letting our expensive GPUs waste their power. As we are processing 1 email at a time, let’s consider this scenario. The email we are processing is an obvious “Spam” email. The model makes a prediction, computes the loss and updates the weights based on this 1 email. Now, the next email comes in. It may look completely different to the first email. So the weights change again. As you can see, processing 1 email at a time gives a very noisy signal on which direction the model moves. So, processing 1 email at a time is inefficient. This leads us to the next option, why not meet somewhere in the middle. How about we process 100 emails at a time. This is pretty much what we call a **batch**. To put it in a more formal definition. A batch is a set of training examples that we process together before performing an update to the model and the size of the batch is called **Batch Size.** So if you look at it, we are not trying to fit the entire dataset into memory and we are also trying to give GPU some amount of work at a time. In **Option 3, **if someone were to ask the question “When are the weights being updated?”, you can clearly answer that right after a batch is processed. This is exactly what is called a **Step**. So, going back to our original training dataset of 100,000 emails. Processing 100 emails at a time would result in 100,000/100 → 1000 steps. There are some exceptions to it but let’s not go there yet. We’ll cross that bridge when we get there. I know I am throwing a lot of terminology but believe me this is the last one. We have identified that processing 100,000 emails 100 at a time results in our model training having 1000 steps. So after 1000 steps, our model should have seen every one of our 100,000 emails. This is exactly what is called an **Epoch** **1 Epoch → 1 complete pass through the training dataset.** You may now ask the question, “isn’t that the end of the model training. I mean once the model processes all the emails, can’t we consider the model trained?” That would be a great question. Answer to that would be, not quite. Processing each email once wouldn’t necessarily mean the model learnt everything because after every step, we only update the weights a little. So after 1 epoch, our model might be better than where it started but doesn’t mean it’s great. So what do we do? We go through the training dataset over and over again. We keep training until we hit an agreed upon condition We randomly picked 100, some of you might have thought 1000. Why not? Larger batches can use more memory. If I process 100 emails at once, I need some memory. If I am processing 1000 or 10,000 even, I need considerably more. So if I increase batch size, at some point, I might run into OOM (Out of memory) issues. Smaller batches waste GPU capacity. As we discussed earlier, smaller batches may not give GPU enough parallel work. As we bought the GPU, we need to keep it busy. As we discussed earlier, smaller batches create noisy updates. Larger batches usually tend to produce stable ones. Of course, there is a lot in here we are not getting into yet. Let’s say that the emails in our training dataset are sorted by “Spam” and “Not Spam”. So, the first 50,000 emails are “Spam” and the rest 50,000 are “Not Spam”. This means, the first 500 batches contain only spam and the next 500 batches contain only not-spam. As you can imagine, this is not a great way to train the model because it won’t be a good model. We’ll get into why in future articles. We want something more random. So, for the first epoch, we randomly shuffle the data and come the next epoch, we shuffle the data again and so on. So we have introduced something which will take time, **Shuffling.** I promise we’ll go into why training data needs to be shuffled in another article. Now that we understand what a batch is, what an epoch is, why we shuffle. Let’s take a look at it from the eyes of the infrastructure. Our pipeline looks something like this. Let’s consider an arbitrary example where our training step takes 50ms and our system takes 100 ms to prepare a batch. So, what happens when a GPU finishes a training step, it waits for the next batch to be fed. As soon as the next batch comes in, it computes the training step again and it waits again. The expensive GPU is sitting idle while it is waiting for the next batch. Phish&Chips is wasting a lot of money making the GPUs sitting idle. Now, we realized that it’s not just enough to have the training data, but we need a system which can This is exactly what “**Data Loading**” is I think I have made this article longer than I planned it out. So let’s save the Data Loading for the next one. [Eye of the Infra — What is a batch and an epoch?](https://pub.towardsai.net/eye-of-the-infra-what-is-a-batch-and-an-epoch-da0cc28464de) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.