{"slug": "tinyml-on-esp32-s3-person-detection-without-sending-anything-to-the-cloud", "title": "TinyML on ESP32-S3: Person Detection Without Sending Anything to the Cloud", "summary": "A developer demonstrates person detection on the ESP32-S3 microcontroller using TinyML, achieving local inference in 80-120 milliseconds without cloud connectivity. The project uses quantized MobileNet models on Visual Wake Words data, with hardware costing $12-15, highlighting privacy, latency, and cost benefits over cloud-based systems.", "body_md": "*Local inference that actually runs.*\n\nYour smart camera is not smart. It's a snitch with a monthly bill.\n\nIt sees a person, panics, compresses a blurry JPEG, uploads your hallway to a data center in Virginia, waits for a GPU to wake up and say \"yeah, that's a person,\" and then charges you $9.99 to tell you what your own eyes could have seen in 100 milliseconds.\n\nWe can do the same job for $12, with no WiFi, no cloud, and no one else ever seeing the pixels. This is how.\n\nI get why we ended up here. Cloud was easy. You slap an RTSP stream on a Pi, send it to Rekognition, done. But for person detection specifically, cloud fails in three predictable, annoying ways.\n\nPrivacy isn't a setting, it's a location. If the frame leaves your house, it's not private. It doesn't matter what the privacy policy says. Local inference means the frame lives for about a tenth of a second in PSRAM and then gets overwritten. The chip doesn't care about your pajamas. It doesn't have a retention policy.\n\nLatency ruins the whole point. Cloud roundtrip is 300ms when your WiFi is happy, two and a half seconds when your microwave is on. An on-device S3 does it in 80 to 120 milliseconds. Your light turns on when you walk in, not after you've already stubbed your toe in the dark.\n\nAnd cost compounds quietly. One camera is \"free tier.\" Five cameras is a business model. The ESP32-S3 draws less than your keyboard backlight and runs on a power bank during a blackout. No API keys, no rate limits, no \"your trial expired\" email at 2am.\n\nIf you need to know *who* the person is, sure, go cloud. If you just need to know *is there a person here right now*, local isn't just cheaper. It's the only design that isn't embarrassing.\n\nForget the old ESP32-CAM. That thing had 520KB of SRAM and the emotional stability of a dying browser tab. You *could* run person detection on it if you liked watching the watchdog timer reboot your board every eight seconds.\n\nThe S3 is a different animal.\n\nDual-core Xtensa LX7 at 240MHz with real vector instructions. 512KB of fast SRAM plus 8MB of PSRAM on every decent devkit. A proper camera interface. And hardware acceleration for the exact thing TinyML does all day: int8 convolutions.\n\nMy workhorses are the XIAO ESP32S3 Sense and the Freenove S3 WROOM CAM. Both have an OV2640 on board, both cost less than lunch. $12 to $15. If you have a bare S3 devkit, add an OV2640 for six bucks.\n\nIt has just enough brain to hold a QVGA frame, run a quantized MobileNet, and still have room to blink an LED smugly when it sees you.\n\nThis is where most people torch their project. They try to port YOLOv8 to a microcontroller to detect 80 classes at 30 FPS. You're not going to.\n\nYou are asking one yes/no question. Human or not human.\n\nThe TinyML classic for this is Visual Wake Words. It's COCO relabeled as person vs no-person. 115,000 images of people being people, and not-people being gloriously boring. That's your starting point.\n\nThe architecture that actually ships is boring on purpose: MobileNetV1 0.25x or MobileNetV2 0.35x, input 96x96 or 160x160, quantized to int8.\n\nWhy this exact combo? Depthwise separable convolutions are cheap. 96x96 is enough to tell a person-shaped blob from a chair-shaped blob at three meters. A 0.25 width multiplier keeps you around 250,000 parameters instead of 25 million. And int8 makes it four times smaller and three times faster on the S3.\n\nFloat version: 1.1MB and completely useless. Int8 version: 285 to 340KB. That fits in flash and leaves room for your actual code. Aim for 85 to 88 percent accuracy on the VWW validation set. If you get 90, you overfit or you're lying to yourself.\n\nTraining is 10% of the work. Deployment is 90% of the suffering. Here is the full loop that actually ships.\n\n**Data.** Don't be a hero and scrape your hallway for two weeks. Start with Visual Wake Words. Then, and this is the cheat code, add 500 images from your own camera. 250 with people, 250 without. Same angle, same lens, same terrible lighting. That tiny bias beats ten thousand more generic COCO images.\n\n**Training.** Boring is good.\n\n``` python\nimport tensorflow as tf\n\nbase = tf.keras.applications.MobileNetV2(\n    input_shape=(96,96,3),\n    alpha=0.35,\n    include_top=False,\n    weights='imagenet'\n)\nbase.trainable = False\n\nmodel = tf.keras.Sequential([\n    base,\n    tf.keras.layers.GlobalAveragePooling2D(),\n    tf.keras.layers.Dropout(0.2),\n    tf.keras.layers.Dense(1, activation='sigmoid')\n])\n\nmodel.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])\nmodel.fit(train_ds, validation_data=val_ds, epochs=5)\n\n# then unfreeze the last 20 layers and fine tune slow\nbase.trainable = True\nfor layer in base.layers[:-20]:\n    layer.trainable = False\n\nmodel.compile(optimizer=tf.keras.optimizers.Adam(1e-5),\n              loss='binary_crossentropy', metrics=['accuracy'])\nmodel.fit(train_ds, validation_data=val_ds, epochs=5)\n```\n\n**Quantization. This is where projects die.**\n\nFloat is for servers. Int8 is for survival. If you skip a representative dataset, your quantized model will confidently think your houseplant is a person. With 99% confidence.\n\n``` python\ndef representative_dataset():\n    for img, _ in train_ds.take(500):\n        yield [img]\n\nconverter = tf.lite.TFLiteConverter.from_keras_model(model)\nconverter.optimizations = [tf.lite.Optimize.DEFAULT]\nconverter.representative_dataset = representative_dataset\nconverter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]\nconverter.inference_input_type = tf.int8\nconverter.inference_output_type = tf.int8\ntflite_model = converter.convert()\n\nopen(\"person_int8.tflite\",\"wb\").write(tflite_model)\n```\n\nCheck accuracy *after* quantization. If it drops more than three percent, your calibration set is bad. Usually it's all white walls.\n\n**To C.** You need a C array for the firmware. The old way is `xxd -i`\n\n. The better way now is keeping it as a partition with esp-ppq. Either works.\n\n**Flash.** With TFLite Micro plus ESP-DL:\n\n```\n#include \"tensorflow/lite/micro/all_ops_resolver.h\"\n#include \"person_model.h\"\n\nconstexpr int kArenaSize = 140 * 1024;\nuint8_t tensor_arena[kArenaSize];\n\ntflite::AllOpsResolver resolver;\nconst tflite::Model* model = tflite::GetModel(person_model);\ntflite::MicroInterpreter interpreter(model, resolver, tensor_arena, kArenaSize);\ninterpreter.AllocateTensors();\n\nwhile(true){\n  camera_fb_t *fb = esp_camera_fb_get();\n  // resize fb->buf from 320x240 to 96x96 RGB into input tensor\n  interpreter.Invoke();\n  int8_t score = interpreter.output(0)->data.int8[0];\n  bool is_person = score > 10; // tune this in real life, not in simulation\n  if(is_person) gpio_set_level(LED_PIN, 1);\n  esp_camera_fb_return(fb);\n}\n```\n\nThat loop gives you 9 to 12 FPS with vanilla TFLM, 15 to 18 FPS with ESP-DL's optimized kernels. No WiFi needed. It just stares and knows.\n\nEveryone shows you accuracy graphs. Nobody shows you the rent bill. Here is the real cost on an S3 with 8MB PSRAM, measured, not marketed.\n\nYour model binary, the int8 MobileNetV2 0.35x at 96x96, sits in flash at around 285 to 340KB. That's fine, flash is cheap.\n\nThe killer is the tensor arena. That's the scratch space the interpreter needs to do math. It *must* live in internal SRAM to be fast. 120 to 180KB. Put it in PSRAM and your FPS collapses to three. This is the single reason the S3 works and the old ESP32 didn't. The S3 has just enough fast RAM to hold the arena and still breathe.\n\nThen you need a frame buffer. QVGA 320x240 in RGB565 is about 150KB. That can live in PSRAM, no problem. You resize it down to 96x96 before inference.\n\nTFLite Micro itself needs about 22KB overhead. Your app logic needs maybe 30KB. If you insist on keeping WiFi and MQTT alive the whole time, add another 80KB and watch your SRAM vanish. Which is why the smartest local designs turn WiFi off completely.\n\nPeak SRAM use ends up around 370KB out of your 512KB. You have about 140KB left. That's it. That's the whole budget. It's tight, it's doable, and it's why you can't be sloppy.\n\nRunning the inference loop flat out at 240MHz dual-core, you're pulling 160 to 190 milliamps at 5V. About 0.8 watts. Warm, not hot. You can run it 24/7 on a 5V 2A brick forever.\n\nThe magic trick is PIR wake. Deep sleep on the S3 is 12 microamps. PIR goes high, you wake, you grab a frame, you run inference, you decide if you need to power the radio. If you only trigger fifty times a day, a 2000mAh LiPo lasts months. Cloud cameras can't do this because they need to keep WiFi associated to be \"smart.\" Your S3 only turns on the radio when it already knows there's a person.\n\nIf you try to run always-on at 10 FPS on battery, that same LiPo dies in four hours. Use a wall wart and stop pretending battery plus always-on vision is easy.\n\nWhere it slaps: battery doorbells that aren't stupid. PIR wakes the board, inference runs, only then does it power WiFi to send a tiny MQTT message: \"yo, person.\" No video upload. Battery lasts a season, not a weekend.\n\nOff-grid stuff: farm gate, warehouse aisle, garage, chicken coop. No internet, still works. Privacy-first rooms: bathroom occupancy without streaming your bathroom to AWS. Elder care that doesn't upload grandma to a bucket. Smart triggers: a mirror that only turns on when a person stands in front of it, a light that ignores your curtains.\n\nWhere it faceplants: crowd counting. It says PERSON, not seven persons. It's binary. Long distance, too. At 96x96, a person at fifteen meters is twelve pixels. It will not see them. Lens and mounting height matter more than your model. Identity? No. It can tell you it's a human-shaped blob, not that it's your mom. For face recognition you need ESP-WHO and even then it's five FPS and you start to hate life. And darkness. The OV2640 is blind in the dark. No IR illuminator means no night vision. AI can't fix physics.\n\nIf your product question is \"is there a human here right now?\" the S3 wins. If your question is \"who are they, what are they holding, and are they on a watchlist?\" you need a Jetson, not a microcontroller.\n\nThe model is twenty percent of the project. The other eighty percent is glue: auto-resizing datasets, batch converting tflite to C, flashing ten boards over serial without losing your mind, logging false positives so you can fix them.\n\nI got tired of rewriting that glue and turned it into field manuals. The whole training to deployment pipeline, from dataset to quantized artifact to OTA, lives in one place now.\n\n**Field manuals from the vault - real links, no filler**\n\nThese are live on **numbpilled.gumroad.com** and they map directly to this build. Grab one seed, pick your route:\n\n**The backbone for this article:**\n\n**Time Necromancy: 100 Python Automations - https://numbpilled.gumroad.com/l/pythonpower**\n\n**Python Automation Secrets - Master Pack - https://numbpilled.gumroad.com/l/masterpython**\n\n**Hardware route, directly relevant to ESP32-S3 person detection:**\n\n**ESP32 Phantom Networks - https://numbpilled.gumroad.com/l/esp32phantom**\n\n**Hardware Signals Field Pack - ESP32 & Sub-GHz RF Research Lab - https://numbpilled.gumroad.com/l/hardwaresignals**\n\n**If you later add physical agents:**\n\n**MasterClaw - https://numbpilled.gumroad.com/l/masterclaw**\n\nBuild one. Point it at your door. Watch the LED flip the instant you walk in with no bars of WiFi. That's when you get why local matters.", "url": "https://wpnews.pro/news/tinyml-on-esp32-s3-person-detection-without-sending-anything-to-the-cloud", "canonical_source": "https://dev.to/numbpill3d/tinyml-on-esp32-s3-person-detection-without-sending-anything-to-the-cloud-ggp", "published_at": "2026-09-03 21:58:43+00:00", "updated_at": "2026-09-03 22:24:38.513635+00:00", "lang": "en", "topics": ["machine-learning", "computer-vision", "ai-tools", "ai-infrastructure"], "entities": ["ESP32-S3", "MobileNet", "Visual Wake Words", "XIAO ESP32S3 Sense", "Freenove S3 WROOM CAM", "OV2640"], "alternates": {"html": "https://wpnews.pro/news/tinyml-on-esp32-s3-person-detection-without-sending-anything-to-the-cloud", "markdown": "https://wpnews.pro/news/tinyml-on-esp32-s3-person-detection-without-sending-anything-to-the-cloud.md", "text": "https://wpnews.pro/news/tinyml-on-esp32-s3-person-detection-without-sending-anything-to-the-cloud.txt", "jsonld": "https://wpnews.pro/news/tinyml-on-esp32-s3-person-detection-without-sending-anything-to-the-cloud.jsonld"}}