Object detection has two broad architectural families: one-stage detectors that predict boxes and classes directly in a single pass, and two-stage detectors that first propose candidate regions, then classify and refine them. Two-stage detectors were the dominant approach from 2014 to 2017 and are still the most accurate choice on many benchmarks when latency isn't the constraint.
The interesting part isn't just that two-stage detectors work — it's how each generation solved a specific, identifiable bottleneck in the previous one. Here's the sequence.
R-CNN (Girshick et al.) was the first detector to apply deep learning effectively to the detection problem. Its pipeline had three independent stages:
This improved detection accuracy by 30% over prior hand-crafted-feature methods (like HOG) on PASCAL VOC, which was a genuinely big deal — it proved CNN features generalize to detection, not just classification.
The cost: running the CNN forward pass 2,000 times per image, once per proposal. That's about 47 seconds per image. Fine for a paper, useless for a product.
The fix here is almost obvious in retrospect: don't run the CNN once per proposal. Run it once on the whole image, and extract each proposal's features from the resulting shared feature map.
The mechanism that makes this possible is RoI Pooling. Since proposals come in arbitrary sizes but downstream fully-connected layers need a fixed-size input, RoI Pooling divides each proposal's region of the feature map into a fixed grid — say 7x7 — and max-pools within each cell. Regardless of how big or small the original proposal was, you get a fixed-length vector out.
feature_map -> RoI Pooling(proposal_region, output_size=7x7) -> fixed_vector -> FC layers -> class + box offsets
This single change made Fast R-CNN 213x faster than R-CNN at test time, and it could be trained end-to-end instead of as three disconnected stages. But Selective Search — a non-learned, CPU-bound algorithm — was still sitting in the pipeline, costing about 2 seconds per image and capping how fast the whole system could go.
Faster R-CNN's contribution is the Region Proposal Network (RPN): a small neural network that shares the backbone's feature map and generates proposals directly, replacing Selective Search entirely.
The RPN works like this: a 3x3 convolution slides over the feature map, feeding two parallel 1x1 convolutions — one producing objectness scores (2k values), one producing box offsets (4k values). At each spatial location, it evaluates k anchor boxes of different scales and aspect ratios (typically k=9: three scales x three ratios), predicting whether each anchor contains an object and how to adjust its coordinates.
Training uses a multi-task loss:
L = L_cls(objectness) + lambda * L_reg(box_offsets)
Anchors are labeled positive if they have IoU > 0.7 with any ground truth box, and negative if IoU < 0.3 with all ground truth boxes. This threshold strategy yields hundreds of high-quality proposals in a single forward pass, no external algorithm required.
The result: a single, fully end-to-end trainable network running at roughly 5 FPS. Not real-time, but orders of magnitude faster than R-CNN's 47 seconds per image, while matching state-of-the-art accuracy.
The two-stage idea didn't stop at Faster R-CNN. Cascade R-CNN extends the pattern to three sequential stages, each trained with a progressively stricter IoU threshold, so detections get refined more precisely at each step.
The underlying tradeoff hasn't changed: two-stage detectors spend more compute deciding where to look before classifying, which costs speed but tends to pay off in accuracy. For applications where real-time inference isn't required and precision matters — medical imaging, satellite imagery, quality inspection — this family of architectures is still a reasonable default.
This is a condensed version of the full lesson, which goes into more detail on anchor design and the RPN training procedure: Two-Stage Detectors, free on NeutralBlock.