Andrej Karpathy's "Deep Dive into LLMs like ChatGPT"

A software engineer learned from Andrej Karpathy's video 'Deep Dive into LLMs like ChatGPT' about the difference between base and chat models, the role of GPUs in training, and what is released when a new model drops. The engineer noted that base models are stochastic text generators that require post-training to become conversational assistants, and that GPUs are essential for parallel computation in training neural networks.

I wanted to dive deeper and learn more about the underlying architecture and function of LLMs. I discovered Andrej Karpathy's YouTube video, "Deep Dive into LLMs like ChatGPT" to be extremely informative and also full of resources to keep learning on my own. As a software engineer, I use Claude Code to write code, generate architecture diagrams, investigate issues, complete Jira tickets, unit test, and for basically all functions of my engineering role. I use ChatGPT to dive deeper into concepts, research topics and improve my writing. But how do these models get to be so smart? How can they be optimized for a specific task, for example coding? What is improved when a new model is released? I wanted to understand more about the inner workings of an LLM so I can make the most of my use of them. Here are some things I learned from this video: The difference between a base model and a chat or instruct model. I learned that fundamentally, base models are stochastic generators of text. The model does not have knowledge of self or function as a chat assistant. It's essentially an autocomplete engine from the internet dataset it was trained on because it's not optimized to interact with humans or provide assistant-like output that we expect in products like ChatGPT. These models need to be post-trained in order to become 'chat-able'. They are fed examples of human conversations and learn over time how to answer or not answer a wide range of prompts. They already have the knowledge of the internet and other training data, but post training, at a basic level, teaches the LLM to be conversational with humans and to communicate information in a useful way. I learned more about the basics of how data centers and GPUs are needed to train the models/neural networks. The neural networks go through training one pass at a time and recompute all of the parameters on each iteration so that the model is better at prediction. During this process, the loss is a number to look out for - the lower the better. The loss should decrease as the model is trained, and therefore predictions from the model will improve. The model will start as a stochastic random token predictor, and eventually become a good predictor of the next token in a sequence, and therefore useful for writing, coding, researching, etc. Why use GPU's instead of CPU's for this training? GPU's can handle parallel computations very efficiently and provide specialization for the types of mathematical computation that training LLMs requires. This is why there's a major demand for data centers, which house hundreds or thousands of GPUs and provide scaleable compute, memory and networking that AI training and inference requires. Note: TPU's were not mentioned but are increasingly used for LLM inference and training, and these are optimized for tensor operations and AI/ML workloads. What is released when a new model drops? We all hear about new models released from companies such as Anthropic, OpenAI, DeepSeek, xAI, etc. But what does this actually encompass? Going back to the difference between a base model and an instruct/chat model, both might be released at the same time for different use cases. Andrej also describes the difference between releases of open source vs closed source models. A release of an open source model will often include the source code for a "forward pass" through the neural network and the parameters of the neural net a list of n billion numbers . For a closed source model, you can use the commercial product for example ChatGPT without knowing the parameters, weights or architecture of the underlying model. You would typically use an API or the company's own user interface to access the model. I learned that models are not inherently good at counting or spelling. As shown in the famous prompt "How many r's are in the word strawberry?", models notoriously answered this question incorrectly. Through the tokenization process, a word such as "strawberry" can be split into several separate tokens, and therefore the LLM has difficulty counting letters as if "strawberry" were a single word. This problem also presents itself with large chunks of text where the model is asked to count occurrences of particular characters or sequences and has trouble accurately doing so. Some of these shortcomings have been solved in modern LLMs with specific training for these situations. Models are not inherently great at exact mathematical computations. Like humans, LLMs are not fetching the answer to a math problem from memory. But unlike humans, who usually apply a mathematical algorithm to solve problems, LLMs are generating responses by predicting the next token from patterns learned during training. Although the models are good at generalizing to new problems, they are not trained for the many steps and precise procedure that computing a solution can require. Again, modern models are better at handling math with enhanced pre and post training on math concepts and reasoning. In general, math is better handled with the use of tools, such as code, that can deterministically solve a problem. Overall, this video gave me the basics of understanding of how these LLMs are built and function under the hood. Karpathy's teaching method broke this down in an easy-to-understand and digestible format. I definitely recommend watching the video. Thanks for reading