The Era of the CNN #
For a long time, Convolutional Neural Networks (CNNs) were the undisputed kings. The logic was straightforward: use filters to scan an image and pick up local features. You start with edges, move to textures, then shapes, and eventually whole objects. I've been looking back at the evolution of these architectures, and the progression is actually pretty logical if you look at the milestones:
LeNet-5 (1998): The OG successful CNN that proved this could work.AlexNet (2012): The massive breakthrough that kicked off the deep learning era.VGGNet (2014): Showed that just making networks deeper helped performance.GoogLeNet (2014): Introduced multi-scale feature extraction.ResNet (2015): A huge deal because skip connections solved the vanishing gradient problem in very deep networks.MobileNet (2017): Focused on making things efficient enough for edge devices.EfficientNet (2019): Optimized the scaling of depth, width, and resolution.ConvNeXt (2022): A modern attempt to bring CNNs back into the fight against Transformers.
The limitation with CNNs is their "local" focus. They are amazing at seeing what's happening in a specific neighborhood of pixels, but they struggle to understand the global context unless the network is incredibly deep. If you have a person on a bike, a CNN might see a wheel and a person separately, but it takes a lot of heavy lifting to realize how they relate spatially across the whole frame.
How Transformers flipped the script #
When Transformers took over NLP, everyone wondered if they could handle vision. The breakthrough came with the Vision Transformer (ViT) in 2020. Instead of sliding a window (convolution) over the image, the ViT treats the image like a sequence of words.
Here is the practical workflow for how a ViT actually processes a frame:
-
Patch Partitioning: Take an image (say 224x224) and chop it into fixed-size patches (like 16x16).
-
Linear Projection: Flatten those patches and turn them into embeddings (vectors).
-
Positional Encoding: Since Transformers don't inherently know "where" things are, you add mathematical markers so the model knows patch A is next to patch B.
-
Self-Attention: This is the magic part. The model uses Query (Q), Key (K), and Value (V) matrices to calculate relationships between every single patch in the image simultaneously.
Instead of asking "what's next to this pixel?", the model asks "how does this patch of a bicycle wheel relate to that patch of a person's hand on the other side of the image?" This global receptive field is what makes ViTs so powerful for complex scene understanding.
If you are looking to implement this, I'd suggest starting with a pre-trained ViT from Hugging Face rather than trying to build the attention mechanism from scratch—it's a massive headache to get the scaling right.
[Next Why most sentiment models fail the nuance test →](/en/threads/8477/)