{"slug": "local-llm-on-esp32-s3-control-a-servo-and-rgb-led", "title": "Local LLM on ESP32-S3 - Control a Servo and RGB LED", "summary": "A Visuino tutorial demonstrates running Needle 2, a 45M-parameter tool-calling model from Cactus Compute, directly on an ESP32-S3 board with 16 MB flash and 8 MB Octal/OPI PSRAM, where the model interprets commands such as \"move servo to 90 degrees\" and returns structured tool calls that C++ firmware validates before driving a servo or the onboard WS2812 RGB LED on GPIO48. The reference port memory-maps the model from flash and uses PSRAM for working memory, and the project reports roughly 47 seconds for a reasoning-on request on its target board, with actual time varying by prompt and output length. The tutorial requires Arduino-ESP32 core 3.3.10, Visuino Professional with the Custom Code component, Chrome or Edge for the local model flasher, and a custom partitions.csv reserving flash from 0x210000 for the model.", "body_md": "Can an ESP32-S3 run a real language model locally? Yes. In this tutorial we run **Needle 2**, a compact 45M-parameter tool-calling model, directly on an ESP32-S3 with **16 MB flash and 8 MB Octal/OPI PSRAM**.\n\nYou will type commands such as:\n\nmove servo to 90 degrees\n\nblink blue 5 times\n\nmove servo to 90 degrees then blink blue 3 times\n\nThe sentence goes **directly to Needle 2**. The Arduino code does not search the sentence for words such as “servo”, “blue”, or “90”. Needle interprets the request and returns structured tool calls. The supplied C++ then validates those structured calls and either sends a normalized value to a Visuino Servo component or controls the onboard RGB LED.\n\nNeedle 2 is an open **45M-parameter** model from Cactus Compute designed for **tool calling, device use, and structured extraction**. The official model is distributed as a compact.cact binary. The ESP32-S3 port used by this tutorial memory-maps the model from flash and uses PSRAM for working memory.\n\nThis makes Needle especially suitable for embedded control. It is not being used here as a general chatbot. Its job is to convert a human request into a small, structured action that firmware can validate and execute.\n\nThe reference ESP32-S3 port targets:\n\n- ESP32-S3, dual core, 240 MHz.\n- **16 MB flash** .\n- **8 MB Octal PSRAM** .\n- An N16R8-class memory configuration.\n- WS2812-compatible onboard RGB LED on **GPIO48** on the reference/tested board.\n\nThe reference project reports roughly **~47 seconds for a reasoning-on request** on its target board; actual time varies with prompt/output length. Long or complex requests can take substantially longer.\n\n- ESP32-S3 board with **16 MB flash + 8 MB Octal/OPI PSRAM** .\n- USB data cable.\n- Small hobby servo.\n- Suitable servo power source, commonly 5 V for small hobby servos.\n- Jumper wires and optional breadboard.\n\n- **[Visuino Professional](https://www.visuino.com/downloads/)** / a Visuino edition with the Custom Code component.\n- Arduino-ESP32 core. This project was tested with **3.3.10** .\n- Desktop web browser **Google Chrome or Microsoft Edge** for the attached local model flasher.\n- The supplied **[Needle2_ESP32S3_Visuino_Files.zip](https://www.visuino.com/Projects/Needle2_ESP32S3_Visuino_Files.zip)** package.\n\nVisuino demo project (Needle2.visuino) file is also available below\n\nStep 3: Install the Needle2 Arduino Library\nExtract the complete ZIP **Needle2_ESP32S3_Visuino_Files.zip.**\n\nCopy this folder:\n\nLibraries\\Needle2\n\ninto:\n\n```\nC:\\Users\\<YOUR_USER>\\Documents\\Arduino\\libraries\\Needle2\n```\n\nAfter copying, this should exist:\n\n```\nC:\\Users\\<YOUR_USER>\\Documents\\Arduino\\libraries\\Needle2\\src\\Needle2.h\n```\n\nand:\n\n```\nC:\\Users\\<YOUR_USER>\\Documents\\Arduino\\libraries\\Needle2\\src\\Needle2UserTools.h\n```\n\nStep 4: Configure the Generated Sketch Folder and Copy Partitions.csv\nBefore we build and upload the project, we need to prepare the custom partition file used by Needle 2.\n\nIn Visuino, open:\n\n**Arduino → Configure**\n\nUnder **Generated sketch directory**, check that the path is:\n\n```\n%userprofile%\\Documents\\Visuino\\Generated\n```\n\nIf this path is already shown, leave it unchanged.\n\nNow open Windows File Explorer and navigate to:\n\n```\n%userprofile%\\Documents\\Visuino\\Generated\n```\n\nIf the **Generated** folder does not exist yet, create it manually.\n\nNext, open the downloaded tutorial files and locate:\n\n```\nVisuino\\partitions.csv\n```\n\nCopy partitions.csv into the **Generated** folder.\n\nBefore the first build, the folder may contain only:\n\nGenerated\\\n\npartitions.csv\n\nLater, when Visuino generates the Arduino sketch, it will also create:\n\nGenerated\\\n\nGenerated.ino\n\npartitions.csv\n\nThe filename must be exactly:\n\npartitions.csv\n\nMake sure Windows has not renamed it to partitions.txt or partitions.csv.txt.\n\nThis custom partition file reserves the flash area used by the Needle 2 model.\n\nThe important model partition begins at:\n\n0x210000\n\nYou only need to copy partitions.csv once, as long as you continue using the same Generated sketch directory.\n\nDo not upload the project yet. We will first finish configuring the Visuino components, then compile and upload everything in the following steps.\n\nStep 5: Start Visuino, and Select the ESP32-S3 Board Type\nOpen Visuino Professional and select:\n\nBoard: ESP32 S3 Development Kit C-1\n\nStep 6: In Visuino Add Custom Code and Servo Component\nAdd these components:\n\n- **Custom Code**\n- **Servo**\n\nThe RGB LED does not need a separate Visuino component in this V26 project; it is controlled by the supplied code.\n\nStep 7: Parse the V26 Arduino Code\nNow we will import the supplied Needle 2 Arduino code into the **Custom Code** component.\n\nRight-click on the **CustomCode1** component.\n\nFrom the popup menu, select:\n\n**Parse Arduino Code**\n\nA window will open where you can paste Arduino code.\n\nOpen the supplied file in Arduino IDE or notepad:\n\nVisuino\\Needle2_Visuino_LED_SERVO_V26.ino\n\nCopy the complete contents of the file and paste them into the **Parse Arduino Code** window.\n\nClick **OK**.\n\nVisuino will analyze the Arduino code and another window will open showing the detected code sections and settings.\n\nClick **OK** again.\n\nVisuino will now automatically configure the Custom Code component with the required sections, including:\n\n- Includes\n- Global Declarations\n- Setup code\n- Loop code\n- Helper functions and other parsed code sections\n\nAfter this step, the Needle 2 code is imported into **CustomCode1** and is ready to be used in the project.\n\nRight-click on the **CustomCode1** component.\n\nFrom the popup menu, select:\n\n**Edit Outputs**\n\nIn the **Outputs window** drag \"**Analog**\" element to the left side\n\nClose the **Outputs window**\n\nConnect:\n\nCustomCode1.Analog1 → Servo1.In\n\nConnect Servo1.Out to the ESP32-S3 pin you want to use for the servo signal. The tested example can use GPIO17.\n\nExample physical wiring:\n\nServo signal → ESP32 GPIO17 pin\n\nServo V+ → ESP32 5V pin\n\nServo GND → supply GND\n\nDo not power a normal hobby servo from the ESP32 3.3 V output.\n\nStep 10: In Visuino Upload the Project\nOn the bottom tab select **Build** and set this settings:\n\nUpload Speed: 921600\n\nUSB Mode: Hardware CDC and JTAG\n\nUSB CDC On Boot: Enabled\n\nUpload Mode: UART0 / Hardware CDC\n\nCPU Frequency: 240MHz (WiFi)\n\nFlash Mode: QIO 80MHz\n\nFlash Size: 16MB (128Mb)\n\nPartition Scheme: No OTA (2MB APP/2MB SPIFFS)\n\nPSRAM: OPI PSRAM\n\nArduino Runs On: Core 1\n\nEvents Run On: Core 1\n\nErase All Flash Before Sketch Upload: Disabled\n\nMake sure the correct port is selected, then click on the \"**Compile/Build and Upload**\" button.\n\nThe model is flashed separately from the Visuino firmware. The package includes a local browser flasher, so users do not need Python, esptool commands, BAT files, or a custom board menu.\n\nFirst close the Visuino Serial Monitor so COM port access is free.\n\nOpen this folder:\n\nNeedle2_Local_Flasher\\\n\nKeep these files together:\n\nindex.html\n\nneedle2.cact\n\nThen:\n\n- Open index.html in desktop **Chrome or Edge** .\n- Click **1. Connect ESP32-S3** .\n- Choose the ESP32-S3 serial port.\n- Click **2. Flash Needle 2** .\n- Wait until the progress reaches 100% and the page says the flash completed successfully.\n\nThe flash address is hard-coded in the page:\n\n0x210000\n\nThe user does not type the address.\n\nThe page also checks the tested model file before flashing:\n\nSize: 13, 737, 807 bytes\n\nSHA256: b43aabfcaf1a6db6acf488076eab71d823c08697c7af4521fc1d174b60ede5ba\n\nIt requests **no full-chip erase**. On the hardware used while preparing this tutorial, flashing the 13.7 MB model took about **7 minutes**; other computers/USB connections may be faster or slower.\n\nThe current tested HTML loads Espressif esptool-js 0.6.1 from a CDN, so the PC needs an Internet connection while opening the flasher. The needle2.cact file itself stays local and is sent from the browser directly to the connected ESP32-S3.\n\nIf Chrome blocks automatic loading of the local needle2.cact, the page displays a **Choose needle2.cact** button. Select the copy in the same flasher folder and continue.\n\nNow we can start the project and check if Needle 2 is running correctly.\n\nReturn to **Visuino** and click the **Play** button.\n\nAt the bottom of the Visuino window, open the **Serial** tab.\n\nMake sure the correct ESP32-S3 COM port is selected.\n\nFor this project, use the same port that was used during upload.\n\nSet the Serial Monitor baud rate to:\n\n115200\n\nThen connect/open the serial port.\n\nOnce the Serial Monitor is connected, press the **RESET / EN** button on the ESP32-S3 board once.\n\nThis restarts the board so you can see the complete startup messages in the Serial Monitor.\n\nWait a few seconds while the ESP32-S3 starts and initializes Needle 2.\n\nIf everything is configured correctly, you should see output similar to:\n\n=== Needle 2 + Visuino FINAL SIMPLE LED + SERVO V26 ===\n\nTools: set_servo + blink_led\n\nServo output: Analog1\n\nReasoning: ON\n\nPlan execution: ATOMIC PREFLIGHT\n\nStarting Needle 2...\n\nmodel mapped: 13762560 bytes\n\nNeedle 2 is ready.\n\nThe most important message is:\n\nNeedle 2 is ready.\n\nThis confirms that the Needle 2 model was found and opened correctly and that the ESP32-S3 is ready to receive natural-language commands.\n\nDo not send a command until you see:\n\nNeedle 2 is ready.\n\nIn the next step, we will send our first command and control the servo.\n\nStep 13: Test RGB LED Control\n\nIn the **Serial Monitor**, type the following command and press **Enter**, or click the **Send** button:\n\nblink blue 5 times\n\nAfter Needle 2 processes the request, the RGB LED on the ESP32-S3 should blink blue 5 times.\n\nNext, try setting how long the LED should blink and how fast:\n\nblink red for 20s 2hz\n\nThe LED should blink red for 20 seconds at 2 Hz.\n\nYou can use these LED colors:\n\n**red, green, blue, yellow, purple, white**\n\nYou can tell Needle 2 how many times to blink the LED, for example:\n\n```\nblink red 5 times\n```\n\nor how long to blink it:\n\n```\nblink red for 20 seconds\n```\n\nYou can also specify the blinking speed:\n\n```\nblink red for 20 seconds at 2hz\n```\n\nBefore any hardware action is performed, the program checks that the generated values are valid and within the allowed limits.\n\n**Tip:** For the most reliable results, use short, clear commands written in lowercase, like the examples above.\n\nIn the **Serial Monitor**, type:\n\n```\nmove servo to 30 degrees\n```\n\nand press **Enter**, or click the **Send** button.\n\n```\nNeedle 2 should generate a set_servo command with the value:\n```\n\ndegrees: 30\n\nVisuino then converts the angle into the normalized value used by the Servo component:\n\n30 / 180 = 0.1667\n\nThe servo should move to approximately 30 degrees.\n\nYou can also try:\n\n```\nmove servo to 0 degrees\nmove servo to 90 degrees\nmove servo to 180 degrees\n```\n\nBefore moving the servo, the program checks that the requested angle is a valid whole number between **0 and 180 degrees**.\n\n**Tip:** For the most reliable results, use short, clear commands written in lowercase.\n\nNow try controlling both the servo and the RGB LED with one command:\n\nmove servo to 90 degrees then blink blue 3 times\n\nNeedle 2 should create two actions in the correct order:\n\n```\nMove the servo to 90 degrees.Blink the RGB LED blue 3 times.\n```\n\nBefore anything is executed, the program checks the complete set of generated actions.\n\nIf one of the actions contains an invalid or unsafe value, the whole command is blocked and no hardware action is performed.\n\nFor the best results, keep combined commands short and simple.\n\nNeedle 2 is a small language model running directly on the ESP32-S3, so shorter commands are usually more reliable than long or complicated instructions.\n\nStep 16: Practical Limits\nNeedle 2 works surprisingly well on the ESP32-S3, but it is still a very small language model running on a microcontroller.\n\nFor the best results:\n\n- Use one action at a time, or a short two-action command.\n- Avoid long or complicated instructions.\n- Use clear values and units, for example: 20s and 2hz.\n- Keep the number of available tools small.\n- Give Needle 2 time to process the request. Some commands can take tens of seconds.\n- Remember that the model can sometimes misunderstand a request.\n\nThe program checks the generated commands before controlling the hardware. If the result is invalid or outside the allowed limits, the action is blocked.\n\nThis combination works well for embedded AI:\n\n**Needle 2 understands the request, while the program checks and safely executes the hardware action.**\n\nThe Needle 2 model normally needs to be flashed only once.\n\nFor future Visuino changes:\n\n- Keep partitions.csv in:\n\n```\n%userprofile%\\Documents\\Visuino\\Generated\n```\n\n- Keep **Erase All Flash Before Sketch Upload = Disabled** .\n- Build and upload the Visuino project normally.\n\nYou do not need to run the HTML model flasher again unless you erase the flash, replace the model, or Needle 2 can no longer find the model.\n\nNormal firmware uploads do not overwrite the Needle 2 model stored at 0x210000.\n\nStep 18: Troubleshooting and Getting Help\nTIf something does not work, first check the **Serial Monitor output** and the Visuino build settings.\n\nMake sure Visuino is configured with:\n\n**PSRAM: OPI PSRAM**\n\nThen build and upload the project again.\n\nnd_model_open failed (-2)\nIf you see something similar to:\n\n```\nERR nd_model_open failed (-2)\n```\n\nthe ESP32-S3 found the model partition, but the Needle 2 model is missing or was not flashed correctly.\n\nClose the Visuino Serial Monitor and run the supplied HTML model flasher again.\n\nDo **not** erase the complete flash.\n\nCheck that the generated sketch directory is:\n\n```\n%userprofile%\\Documents\\Visuino\\Generated\n```\n\nand that this folder contains:\n\n```\npartitions.csv\n```\n\nThe file must be there before building the project.\n\nAnalog1 was not declared\nMake sure the Custom Code component contains an **Analog output** named exactly:\n\nAnalog1\n\nThen parse the supplied V26 Arduino code again.\n\nServo does not move\nCheck:\n\n- CustomCode1.Analog1 is connected to Servo1.In.\n- The correct servo output pin is selected.\n- The servo has a suitable power supply.\n- The servo and ESP32-S3 share a common ground.\n\nThen test:\n\n```\nmove servo to 90 degrees\n```\n\nRGB LED does not work\nThis tutorial uses the onboard addressable RGB LED on **GPIO48** of the tested ESP32-S3 board.\n\nSome ESP32-S3 boards use a different LED pin or different LED hardware.\n\n```\nBrowser flasher cannot connect\n```\n\nClose the Visuino Serial Monitor and any other program using the ESP32-S3 COM port.\n\nThen reconnect the USB cable, reload index.html in desktop Chrome or Edge, and try again.\n\nI erased the complete flash\nUpload the Visuino project again and then run the Needle 2 HTML model flasher again.\n\nAfterward, remember to keep:\n\n**Erase All Flash Before Sketch Upload: Disabled**\n\nUse short, clear commands written in lowercase.\n\nFor example:\n\n```\nmove servo to 90 degreesblink blue 5 times\n```\n\nNeedle 2 is a small embedded model and can sometimes misunderstand long or complicated requests.\n\nThe V26 program checks the generated actions before controlling the hardware, so invalid actions can be blocked instead of executed.\n\nHow to Ask ChatGPT for Help\nIf you still cannot solve the problem, provide ChatGPT with:\n\n- the complete **build/upload log**\n- the complete **Serial Monitor output**\n- a screenshot of the Visuino settings or wiring if relevant\n\nYou can use this prompt:\n\nI am following the ESP32-S3 + Needle 2 + Visuino V26 tutorial.\n\nHardware:\n\n- ESP32-S3\n- 16 MB flash\n- 8 MB OPI/Octal PSRAM\n\nVisuino settings:\n\n- Flash Size: 16MB\n- Partition Scheme: No OTA\n- PSRAM: OPI PSRAM\n- Erase All Flash Before Sketch Upload: Disabled\n- Generated sketch directory: %userprofile%\\Documents\\Visuino\\Generated\n\nThe supplied partitions.csv is in the Generated folder.\n\nThe Needle 2 model was flashed using the supplied HTML flasher.\n\nI am using Needle2_Visuino_LED_SERVO_V26.ino.\n\nPlease diagnose the problem from the logs below and give me the smallest thing to check first.\n\nBUILD / UPLOAD LOG:\n\n[paste the complete log]\n\nSERIAL MONITOR OUTPUT:\n\n[paste the complete output]\n\nStep 19: Adding New Needle 2 Tools Safely\nOnce the basic project is working, you can add your own Needle 2 tools.\n\nA new tool needs two parts:\n\n- A tool definition in:\n\nNeedle2UserTools.h\n\n- Matching validation and hardware-control code in the Arduino program.\n\nThe tool definition tells Needle 2:\n\n- the tool name\n- what the tool does\n- which values it can generate\n\nThe Arduino code then checks those values and performs the actual hardware action.\n\nFor example, adding a tool definition called:\n\nturn_relay\n\ndoes not automatically control a relay. You must also add code that recognizes turn_relay, validates its arguments, and safely controls the relay output.\n\nKeep Tools Simple\nNeedle 2 is a small embedded model, so keep tool definitions short and clear.\n\nFor best results:\n\n- Add one new tool at a time.\n- Use simple tool names.\n- Keep the number of arguments small.\n- Use clear limits for numeric values.\n- Validate every generated value before controlling hardware.\n- Keep the total number of tools reasonably small.\n\nAfter adding a new tool, rebuild and upload the Visuino project and test it with short, simple commands first.\n\nDo not rely only on Needle 2 to decide what is safe. The Arduino code should always remain responsible for checking values before any hardware action is performed.\n\nStep 20: FAQDoes Needle 2 need Wi-Fi while controlling the servo or LED?\nNo. After the model and firmware are installed, inference runs locally on the ESP32-S3. No cloud API or PC inference server is used.\n\nDoes the local HTML flasher mean the LLM runs in Chrome?\nNo. Chrome is used only as a convenient one-time ESP32 model flasher. Needle inference runs on the ESP32-S3 afterward.\n\nDoes the flasher create the partition table?\nNo. partitions.csv creates the custom model partition during the Arduino/Visuino firmware build. The HTML flasher only writes needle2.cact to 0x210000.\n\nWhy select No OTA if we use a custom partition table?\nIt gives Visuino a normal built-in board-menu selection. Arduino-ESP32's build system gives precedence to a partitions.csv placed in the sketch folder, so the supplied CSV becomes the actual partition layout for this project.\n\nDo I need the old Needle2 16MB partition menu entry?\nNo. This updated tutorial intentionally avoids it.\n\nDo I need Python or a BAT file?\nNo for the normal tutorial flow. The supplied local HTML flasher handles the model upload in Chrome/Edge.\n\nDo I have to flash the 13.7 MB model after every code upload?\nNo. Flash it once, then keep full-chip erase disabled. Reflash only after a full erase, a model change, or a model-open failure.\n\nCan I add more tools?\nYes, but keep the schema compact and add matching deterministic C++ validation/execution for each new tool.\n\nStep 21: Model License and Redistribution\nThe bundled needle2.cact model is from Cactus Compute's Needle 2 release and is distributed under the **Apache License 2.0**. Keep the included license and attribution/third-party notice files with redistributed copies of the package/model.\n\nThe surrounding tutorial can use a different content license; the model remains under its own Apache-2.0 license.\n\nStep 22: Conclusion\nYou now have a genuine local tool-calling language model on an ESP32-S3 interpreting natural-language requests and controlling hardware through Visuino.\n\nThe important architecture is simple:\n\nNeedle 2 = language understanding\n\nC++ = bounded validation and execution\n\nVisuino = visual hardware integration\n\nThe updated setup also removes the most awkward parts of the older process: there is no custom partition menu to install and no Python/BAT model-flashing command. Users select the standard **No OTA** option, keep the supplied partitions.csv in the configured Generated sketch directory, and flash the model from a local Chrome/Edge page.\n\n- Needle 2 model: [https://huggingface.co/Cactus-Compute/needle2](https://huggingface.co/Cactus-Compute/needle2)\n- Needle 2 ESP32-S3 port: [https://github.com/andrisgauracs/needle-2-esp32](https://github.com/andrisgauracs/needle-2-esp32)\n- Arduino-ESP32 custom partition tables: [https://docs.espressif.com/projects/arduino-esp32/en/latest/tutorials/partition_table.html](https://docs.espressif.com/projects/arduino-esp32/en/latest/tutorials/partition_table.html)\n- Espressif esptool-js: [https://github.com/espressif/esptool-js](https://github.com/espressif/esptool-js)\n- Visuino: [https://www.visuino.com/](https://www.visuino.com/)\n\n[Read more](<javascript:void(0)>)", "url": "https://wpnews.pro/news/local-llm-on-esp32-s3-control-a-servo-and-rgb-led", "canonical_source": "https://www.hackster.io/ronfrtek/local-llm-on-esp32-s3-control-a-servo-and-rgb-led-0c8753", "published_at": "2026-11-16 14:58:34+00:00", "updated_at": "2026-11-16 15:20:22.132709+00:00", "lang": "en", "topics": ["large-language-models", "ai-tools", "ai-agents", "artificial-intelligence"], "entities": ["Needle 2", "Cactus Compute", "ESP32-S3", "Visuino", "Arduino-ESP32", "Visuino Professional"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/local-llm-on-esp32-s3-control-a-servo-and-rgb-led", "markdown": "https://wpnews.pro/news/local-llm-on-esp32-s3-control-a-servo-and-rgb-led.md", "text": "https://wpnews.pro/news/local-llm-on-esp32-s3-control-a-servo-and-rgb-led.txt", "jsonld": "https://wpnews.pro/news/local-llm-on-esp32-s3-control-a-servo-and-rgb-led.jsonld"}}