Local inference that actually runs.
Your smart camera is not smart. It's a snitch with a monthly bill.
It sees a person, panics, compresses a blurry JPEG, uploads your hallway to a data center in Virginia, waits for a GPU to wake up and say "yeah, that's a person," and then charges you $9.99 to tell you what your own eyes could have seen in 100 milliseconds.
We can do the same job for $12, with no WiFi, no cloud, and no one else ever seeing the pixels. This is how.
I get why we ended up here. Cloud was easy. You slap an RTSP stream on a Pi, send it to Rekognition, done. But for person detection specifically, cloud fails in three predictable, annoying ways.
Privacy isn't a setting, it's a location. If the frame leaves your house, it's not private. It doesn't matter what the privacy policy says. Local inference means the frame lives for about a tenth of a second in PSRAM and then gets overwritten. The chip doesn't care about your pajamas. It doesn't have a retention policy.
Latency ruins the whole point. Cloud roundtrip is 300ms when your WiFi is happy, two and a half seconds when your microwave is on. An on-device S3 does it in 80 to 120 milliseconds. Your light turns on when you walk in, not after you've already stubbed your toe in the dark.
And cost compounds quietly. One camera is "free tier." Five cameras is a business model. The ESP32-S3 draws less than your keyboard backlight and runs on a power bank during a blackout. No API keys, no rate limits, no "your trial expired" email at 2am.
If you need to know who the person is, sure, go cloud. If you just need to know is there a person here right now, local isn't just cheaper. It's the only design that isn't embarrassing.
Forget the old ESP32-CAM. That thing had 520KB of SRAM and the emotional stability of a dying browser tab. You could run person detection on it if you liked watching the watchdog timer reboot your board every eight seconds.
The S3 is a different animal.
Dual-core Xtensa LX7 at 240MHz with real vector instructions. 512KB of fast SRAM plus 8MB of PSRAM on every decent devkit. A proper camera interface. And hardware acceleration for the exact thing TinyML does all day: int8 convolutions.
My workhorses are the XIAO ESP32S3 Sense and the Freenove S3 WROOM CAM. Both have an OV2640 on board, both cost less than lunch. $12 to $15. If you have a bare S3 devkit, add an OV2640 for six bucks.
It has just enough brain to hold a QVGA frame, run a quantized MobileNet, and still have room to blink an LED smugly when it sees you.
This is where most people torch their project. They try to port YOLOv8 to a microcontroller to detect 80 classes at 30 FPS. You're not going to.
You are asking one yes/no question. Human or not human.
The TinyML classic for this is Visual Wake Words. It's COCO relabeled as person vs no-person. 115,000 images of people being people, and not-people being gloriously boring. That's your starting point.
The architecture that actually ships is boring on purpose: MobileNetV1 0.25x or MobileNetV2 0.35x, input 96x96 or 160x160, quantized to int8.
Why this exact combo? Depthwise separable convolutions are cheap. 96x96 is enough to tell a person-shaped blob from a chair-shaped blob at three meters. A 0.25 width multiplier keeps you around 250,000 parameters instead of 25 million. And int8 makes it four times smaller and three times faster on the S3.
Float version: 1.1MB and completely useless. Int8 version: 285 to 340KB. That fits in flash and leaves room for your actual code. Aim for 85 to 88 percent accuracy on the VWW validation set. If you get 90, you overfit or you're lying to yourself.
Training is 10% of the work. Deployment is 90% of the suffering. Here is the full loop that actually ships.
Data. Don't be a hero and scrape your hallway for two weeks. Start with Visual Wake Words. Then, and this is the cheat code, add 500 images from your own camera. 250 with people, 250 without. Same angle, same lens, same terrible lighting. That tiny bias beats ten thousand more generic COCO images.
Training. Boring is good.
import tensorflow as tf
base = tf.keras.applications.MobileNetV2(
input_shape=(96,96,3),
alpha=0.35,
include_top=False,
weights='imagenet'
)
base.trainable = False
model = tf.keras.Sequential([
base,
tf.keras.layers.GlobalAveragePooling2D(),
tf.keras.layers.Dropout(0.2),
tf.keras.layers.Dense(1, activation='sigmoid')
])
model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])
model.fit(train_ds, validation_data=val_ds, epochs=5)
base.trainable = True
for layer in base.layers[:-20]:
layer.trainable = False
model.compile(optimizer=tf.keras.optimizers.Adam(1e-5),
loss='binary_crossentropy', metrics=['accuracy'])
model.fit(train_ds, validation_data=val_ds, epochs=5)
Quantization. This is where projects die.
Float is for servers. Int8 is for survival. If you skip a representative dataset, your quantized model will confidently think your houseplant is a person. With 99% confidence.
def representative_dataset():
for img, _ in train_ds.take(500):
yield [img]
converter = tf.lite.TFLiteConverter.from_keras_model(model)
converter.optimizations = [tf.lite.Optimize.DEFAULT]
converter.representative_dataset = representative_dataset
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]
converter.inference_input_type = tf.int8
converter.inference_output_type = tf.int8
tflite_model = converter.convert()
open("person_int8.tflite","wb").write(tflite_model)
Check accuracy after quantization. If it drops more than three percent, your calibration set is bad. Usually it's all white walls.
To C. You need a C array for the firmware. The old way is xxd -i
. The better way now is keeping it as a partition with esp-ppq. Either works.
Flash. With TFLite Micro plus ESP-DL:
#include "tensorflow/lite/micro/all_ops_resolver.h"
#include "person_model.h"
constexpr int kArenaSize = 140 * 1024;
uint8_t tensor_arena[kArenaSize];
tflite::AllOpsResolver resolver;
const tflite::Model* model = tflite::GetModel(person_model);
tflite::MicroInterpreter interpreter(model, resolver, tensor_arena, kArenaSize);
interpreter.AllocateTensors();
while(true){
camera_fb_t *fb = esp_camera_fb_get();
// resize fb->buf from 320x240 to 96x96 RGB into input tensor
interpreter.Invoke();
int8_t score = interpreter.output(0)->data.int8[0];
bool is_person = score > 10; // tune this in real life, not in simulation
if(is_person) gpio_set_level(LED_PIN, 1);
esp_camera_fb_return(fb);
}
That loop gives you 9 to 12 FPS with vanilla TFLM, 15 to 18 FPS with ESP-DL's optimized kernels. No WiFi needed. It just stares and knows.
Everyone shows you accuracy graphs. Nobody shows you the rent bill. Here is the real cost on an S3 with 8MB PSRAM, measured, not marketed.
Your model binary, the int8 MobileNetV2 0.35x at 96x96, sits in flash at around 285 to 340KB. That's fine, flash is cheap.
The killer is the tensor arena. That's the scratch space the interpreter needs to do math. It must live in internal SRAM to be fast. 120 to 180KB. Put it in PSRAM and your FPS collapses to three. This is the single reason the S3 works and the old ESP32 didn't. The S3 has just enough fast RAM to hold the arena and still breathe.
Then you need a frame buffer. QVGA 320x240 in RGB565 is about 150KB. That can live in PSRAM, no problem. You resize it down to 96x96 before inference.
TFLite Micro itself needs about 22KB overhead. Your app logic needs maybe 30KB. If you insist on keeping WiFi and MQTT alive the whole time, add another 80KB and watch your SRAM vanish. Which is why the smartest local designs turn WiFi off completely.
Peak SRAM use ends up around 370KB out of your 512KB. You have about 140KB left. That's it. That's the whole budget. It's tight, it's doable, and it's why you can't be sloppy.
Running the inference loop flat out at 240MHz dual-core, you're pulling 160 to 190 milliamps at 5V. About 0.8 watts. Warm, not hot. You can run it 24/7 on a 5V 2A brick forever.
The magic trick is PIR wake. Deep sleep on the S3 is 12 microamps. PIR goes high, you wake, you grab a frame, you run inference, you decide if you need to power the radio. If you only trigger fifty times a day, a 2000mAh LiPo lasts months. Cloud cameras can't do this because they need to keep WiFi associated to be "smart." Your S3 only turns on the radio when it already knows there's a person.
If you try to run always-on at 10 FPS on battery, that same LiPo dies in four hours. Use a wall wart and stop pretending battery plus always-on vision is easy.
Where it slaps: battery doorbells that aren't stupid. PIR wakes the board, inference runs, only then does it power WiFi to send a tiny MQTT message: "yo, person." No video upload. Battery lasts a season, not a weekend.
Off-grid stuff: farm gate, warehouse aisle, garage, chicken coop. No internet, still works. Privacy-first rooms: bathroom occupancy without streaming your bathroom to AWS. Elder care that doesn't upload grandma to a bucket. Smart triggers: a mirror that only turns on when a person stands in front of it, a light that ignores your curtains.
Where it faceplants: crowd counting. It says PERSON, not seven persons. It's binary. Long distance, too. At 96x96, a person at fifteen meters is twelve pixels. It will not see them. Lens and mounting height matter more than your model. Identity? No. It can tell you it's a human-shaped blob, not that it's your mom. For face recognition you need ESP-WHO and even then it's five FPS and you start to hate life. And darkness. The OV2640 is blind in the dark. No IR illuminator means no night vision. AI can't fix physics.
If your product question is "is there a human here right now?" the S3 wins. If your question is "who are they, what are they holding, and are they on a watchlist?" you need a Jetson, not a microcontroller.
The model is twenty percent of the project. The other eighty percent is glue: auto-resizing datasets, batch converting tflite to C, flashing ten boards over serial without losing your mind, logging false positives so you can fix them.
I got tired of rewriting that glue and turned it into field manuals. The whole training to deployment pipeline, from dataset to quantized artifact to OTA, lives in one place now.
Field manuals from the vault - real links, no filler
These are live on numbpilled.gumroad.com and they map directly to this build. Grab one seed, pick your route:
The backbone for this article:
Time Necromancy: 100 Python Automations - https://numbpilled.gumroad.com/l/pythonpower
Python Automation Secrets - Master Pack - https://numbpilled.gumroad.com/l/masterpython
Hardware route, directly relevant to ESP32-S3 person detection:
ESP32 Phantom Networks - https://numbpilled.gumroad.com/l/esp32phantom
Hardware Signals Field Pack - ESP32 & Sub-GHz RF Research Lab - https://numbpilled.gumroad.com/l/hardwaresignals
If you later add physical agents:
MasterClaw - https://numbpilled.gumroad.com/l/masterclaw
Build one. Point it at your door. Watch the LED flip the instant you walk in with no bars of WiFi. That's when you get why local matters.