TinkerSight: An AI Friend for When You're Stuck With a Device

A developer built TinkerSight, a vision-first AI technical guide that lets users photograph an unfamiliar appliance or device and receive step-by-step guidance grounded in official manufacturer documentation. The system runs the open-weight Qwen3-VL 2B vision-language model locally via Ollama behind a React/Vite frontend and FastAPI backend, and includes a deterministic safety layer that halts or asks for clarification on hazards such as exposed wiring, gas leaks, smoke, or sparks. The developer said the design principle was "don't pretend to know something that isn't visible or verified.

This is a submission for the Hacktoberfest Weekend Challenge: Build for a Friend https://dev.to/challenges/hacktoberfest-weekend-2026-10-01 I built TinkerSight , a vision-first AI technical guide for my friend who lives alone and sometimes struggles with unfamiliar appliances and everyday devices. The problem is simple: when you're standing in front of an unfamiliar AC remote, washing machine, or device, the manual often isn't written for the one question you actually have. TinkerSight lets someone show a device through a photo, explain what they want to do, and get simple, step-by-step guidance. Its core idea is: See → Ground → Simplify → Guide It doesn't just try to recognize what's in the image. When possible, TinkerSight retrieves relevant official manufacturer documentation and uses it to ground the guidance. If the image is unclear or the situation could be dangerous, it asks for clarification or stops instead of confidently guessing. I built TinkerSight because I wanted to create something that felt less like searching through a manual and more like having someone standing beside you when you need help. Here is a short demo showing three situations: Demo video: Watch the TinkerSight Demo video: https://1drv.ms/v/c/bbab0edd6634b417/IQDjln7aXVbrR5HvUco5p-YwAXIP2aJejHEnnWeCJaIsJGc?e=KX55aw https://1drv.ms/v/c/bbab0edd6634b417/IQDjln7aXVbrR5HvUco5p-YwAXIP2aJejHEnnWeCJaIsJGc?e=KX55aw The complete source code is available on GitHub: https://github.com/aisha453/TinkerSight https://github.com/aisha453/TinkerSight The repository includes the React frontend, FastAPI backend, local AI integration, manufacturer-document grounding, and safety handling. TinkerSight is built around Qwen3-VL 2B , an open-weight vision-language model running locally through Ollama . The architecture is: React + Vite → FastAPI → Ollama → Qwen3-VL 2B The user uploads an image and describes what they want to do. The vision model first identifies the device, visible controls, brand information, and relevant context. For supported manufacturers, TinkerSight then retrieves information from selected official manufacturer webpages and uses that information to ground the response. I also added a deterministic safety layer for obvious hazardous situations such as exposed electrical wiring, gas leaks, smoke, sparks, burning smells, and internal appliance repairs. One important design principle was: don't pretend to know something that isn't visible or verified. For example, TinkerSight does not claim an exact appliance model unless it can actually read the model information. Open innovation made this project possible in a way that would have been difficult to achieve with a closed API alone. Because TinkerSight uses an open-weight vision-language model through local inference, the AI can run on the user's own machine without sending every image to a proprietary vision API. It also means the model layer can be replaced or improved without rebuilding the entire application. More importantly, using open AI made it possible for me to experiment with the behavior of the system itself: how it interprets images, how it handles uncertainty, how manufacturer information is introduced, and when it should refuse to provide instructions. For a tool dealing with someone's home, appliances, and potentially sensitive images, having the ability to run the AI locally is especially meaningful. No partner prize category claimed. TinkerSight currently uses open-source/open-weight AI and local inference, but I did not add a partner technology solely to qualify for a prize category.